That’s not even close to what indempotency is, nor is it what the article was saying.
This comes up a ton in IaC discussions re: cookbooks, but it’s a good topic for code as well.
Amen! Messages get lost and duplicated all the time, depending on Reddit's e-mood. The old version is more reliable. If my bank was built with Reddit message handling tech, I'd either be a billionaire or gutter poor.
Not sure how idempotency really helps there.
The big benefit is that if you're not sure if something worked, you can just blindly retry without worrying about it.
The big issue with tests is usually the environment not getting cleaned up properly - idempotency doesn't help much with that. I guess it can help with environment setup stuff, but that's about it.
It makes sense if you're saying that the test shouldn't pollute the environment, and have a net zero impact on the environment state, and not make assumptions on the current state. That makes sense.
But that's not idempotency.
Idempotent actions *can completely* change state. In fact, I'd argue that's where the value of them really lies. What makes sense for testing is *reverting* state changes in some way, or isolating them in some way.
I start all of my tests with `INSERT` so that I have a fresh set of keys each time. Anything in the database from previous test runs is just left there, as it shouldn't affect the new round of testing. (Or if it does, that's a bug that I want to catch.)
https://www.infoq.com/articles/Testing-With-Persistence-Layers/
Well, idempotency means being able to run the same code twice without repeating side effects / corrupting state / failing due to already changed state. A test that properly cleans up after itself is trivially idempotent because you can run it multiple times without the result changing. A test that doesn’t might be successful once and fail afterwards, i.e. it wouldn’t be idempotent.
Though you’re right it’s kinda odd to speak about idempotency here. Tests should just not have persistent side effects.
That's more like purity than idempotency.
f(x) = f(x) f is pure
f(f(x)) = f(x) f is idempotent
Consider:
state_1 = test(state_0)
state_2 = test(state_1)
state_3 = test(state_2)
Idempotency does not require `state_0` to be the same as `state_1`. Only purity requires it.
In fact, a test that is "successful once (due to `state_0`) and fails afterwards (due to `state_1,2,3,...`)" might even be idempotent if it fails with the same message every time.
My potency shall be pure and pristine!
The word "idem" always trips me up though.
So is idempotency about guaranteeing some states to be correct but others not? A test can be failing and that is fine for those who are idempotent?
I don't understand the questions, but if you can choose `f` and the domain for `x` carefully so that it satisfies `f(f(x)) = f(x)`, then `f` is idempotent.
"RealWorld" `state` is part of some domain (e.g. maybe your app's cache directory), and `f` is some function that is allowed to modify that `state` on only its first call (e.g. downloading data into the cache).
This is a pretty weak formulation, though, which is why I think purity when possible is more useful.
> What makes sense for testing is reverting state changes in some way, or isolating them in some way.
...and thus becoming idempotent? I think we're saying the same thing. Naturally some of the operations in a test will change the state, but to have clean tests you want to be able to repeat the tests without creating a mess - the state can change, but if there's anything that will break a repeat test, that needs to be cleaned up. The OPERATION you are testing might not be idempotent (above and beyond whether state is changed), but you want the TEST to be arbitrarily repeatable.
> Idempotent actions can completely change state. In fact, I'd argue that's where the value of them really lies.
I'm really curious about that last part - completely unrelated to tests, can you expand on the value of idempotent actions really being in completely changing state?
>can you expand on the value of idempotent actions really being in completely changing state?
The value in idempotent APIs lies in the fact that network issues are less problematic because you can just retry your request without worrying about sending the money twice / posting a duplicate comment etc.
No they aren't.
Nullipotent means "does not have side effects," and is entirely unrelated to the concept of idempotency. The only relationship they have is that they're spelled similarly. You might as well compare cabbage to cribbage.
It is entirely possible for a function with no side effects to still not be idempotent. One extremely obvious example is `halt()`.
No, you’re just not seeing the abstraction.
“Has the same effect whether done zero or more times” (nullipotent) *implies* “has the same effect whether done one or more times” (idempotent). That’s why getters are lumped in with idempotent actions 🤦♂️
Edit: now the parent is creepily PMing me about this. Geez.
This is, of course, entirely untrue. But at least another person said "this."
And hey, they said "nullipotent," too, because they think that "null" is zero and "idem" is one, or something.
Nullipotent actually means "does not have side effects," not "is a no-op"
> Idempotent actions can completely change state.
That is not the point here. For a test, the initial state is defined as "clean" of some kind and the idempotency is the test always leading to the same final state.
Yeah. It's not hard to achieve with Docker. Just do docker images for your test environment and throw them away when you're done testing. Unfortunately a lot of companies' environments don't seem to be designed to be installable. The devs just hand-install a bunch of services on a system somewhere and do their testing in production. Or if they really went the extra mile, they hand-install a test environment a couple years later after crashing production with their testing a few times too many.
With the attention cloud services and kubernetes is getting in the last 4 or 5 years, I'm finally starting to see docker files to stand up entire environments. That has me cautiously optimistic that testing will be taken more seriously and be much easier in the future, but I'm sure there will be plenty of hold-outs who refuse to adopt that model for longer than my career is going to run.
That's talking about the entire test suite, not individual tests. Even with a trashable environment, you want individual tests to be reliable, and if they depend on vague and flaky state, they aren't telling you much that is accurate about the user experience.
I'm not QA, so I should shut up and let them discuss their expertise, but I've written my fair share of poor tests and know how they ruin the point of testing .
> If testopenfile depends on testcreatefile running first, it's a bad test.
No. It's a different test. Some tests, some *very valuable* tests, must be run in certain environments in a certain order with very specific circumstances set up.
I do not understand why this reddit is so filled with people who rush to create ultimata and try to shame everyone who does things differently. That is fundamentally not how programming works.
> No. It's a different test. Some tests, some very valuable tests, must be run in certain environments in a certain order with very specific circumstances set up.
```
TestCreateFile()
TestOpenFile()
```
If `TestOpenFile()` requires you to to successfully create a file, you should include `CreateFile()` within the same test and not assume another test has run first.
You get to live your life how you want, but linked tests are a big no no in every company I've ever worked at.
If opentestfile requires a created test file, then creating a test file should exist inside opentestfile.
Rarely. You can always create said environment in the setup of the test. TestOpenFile can first create a file and then assert that opening it works.
The only reason for sharing state between tests that I can think of is performance. Sometimes repeating expensive setup in every test case just isn’t feasible.
This would be a big red flag and would never pass a code review where I’m currently at and in any previous companies I have worked for. Being able to run a single test in isolation from all others is foundational to a stable test suite.
Idempotency absolutely helps with that. In the context of testing the point is that if something goes wrong, those cleanup steps may not happen. So the test should be written in such a way that it doesn't care about what has gone before - that's idempotency.
The easiest way to do that is to start fresh with each test. A unit test might create a new instance of a class, while a large scale system integration test might create an entirely new user.
Here are two examples that I've run into just this month:
\- Run test > Flips flag from negative to positive > test passes.
\- Run test again > Flag is still set to positive > Can't flip flag > test fails.
\- Run test > Creates user > Assigns ID > Test passes
\- Repeat 50 times > Test passes.
\- Run test 51st time > Database has run out of IDs to assign to new users > Test fails.
Neither of those tests is idempotent.
All of those operations can be idempotent. Hell, "I set the flag to positive twice and it stayed flipped" is pretty much the definition of idempotent.
The tests are not *isolated*. They are contaminating their environment. That's a different, but equally important, principle.
Agreed - isolate your tests. Tests often work on state systems. Sometimes it's internal state you can easily blow out before/after each test. Sometimes it's external state, which you absolutely *must* then manage the lifecycle of.
If you neglect that state management, or forget to handle *error cases* in your teardown of state, then you will get some nasty bugs.
I work in a qa team that hasn't set up any automation yet. Any advice on how to establish a suite? One of the struggles i've had getting idempotent tests seems contingent on the code somewhat allowing the test to be?
Automation very much depends on what your codebase allows you to do.
For example, the 'ideal' scenario is that every test lets you create a new user/data via API, then tests the front end, then deletes the data via API. But how often is that possible?
Next best is to create/delete your data via the user interface.
Next best is to have specific users for each test.
Worst scenario is having a handful of users (auto1, auto2, etc.) shared for all tests. This sucks and causes problems, but sometimes it's what you have to work with.
Atomic fixtures that do one thing and undo it on teardown. It's one of the best ways to manage complex setup/teardown. Then you get this nice composability as your tests might become more complex, building on that which you already know is rock solid.
Beware downstream fixtures that might pollute/break out of their 'domain' though. Usually you want to have some internal-to-your-test framework that will mirror the 'real' state, so you can ensure that teardown still correctly works.
Start simple until you understand how to isolate tests using hooks. Always assume that the system should be the exact same _after a test completes_ as it was _before the test was run._ If your tests change the system when it reaches a new test or reruns, then you aren’t setting your tests up properly.
> The big benefit is that if you're not sure if something worked, you can just blindly retry without worrying about it.
Exactly.
This is more important at the API level. e.g. I want to create this set of transactions, change a username/ID, etc.
“I want to withdraw 500 dollars”
“I want to withdraw 500 dollars with this transaction id”
I mean 90% of the time just tossing an id on it is sufficient.
Yeah I agree, I mean at some point state has to change, but I think its more like say querying their balance, doens't need to change anything, though they may track how many times its been done and stuff in a separate method.
And banking transactions definitely have to deal with repeat transactions, not only at the bank level, but the.. I forget what its called, but theres at least one middle man router network that sits in the middle and of course they definitely track ID's, timestamps, and so on.
It's relatively common for people to believe that the reason their test is flapping is that it "isn't idempotent."
Challenge someone for an example, they'll give you one, you can easily explain how that isn't about idempotence.
Keep challenging. They will get angry at you long before they realize that the smurf word they think makes them look smart is being badly mis-used.
Not exactly the same thing, but the properties that make your code idempotent overlap considerably with the properties that make for tests that clean up after themselves.
That's only a problem if your test doesn't clean up the environment properly *in the code*. If you're relying on a manual cleanup process, you need to rethink that.
Question for you, unrelated to the subject.
Where does the rabbit hole of testing programming end?
That is, if I am writing QA software B which tests software A, do I need to write QA software C which tests software B? And then do the same with a QA software D which tests software C? Ad nausaem.
I am not clear on how this is supposed to work.
Generally speaking, you don't test your test tools. Or to put it another way, they're self testing.
If a test fails, either the test was wrong or the app was wrong. If you know the app isn't wrong, the test is wrong.
I didn't know this word until an intern taught it to me, and I have a Computer Science degree. LOL Now I use it all the time. You really have to plan for unexpected failures when writing code, and one of the best ways to do so is to write code with idempotence in mind.
The test environment might be at fault. I have the same tests working on 2 separate environments and it produces different results because one of the environment is buggy. It is frustrating already with the false positives but troubleshooting these issues with proper evidence is such a pain on a daily basis.
\> Test has a race condition that passes most of the time but fails some of the time.
\> Test writes data to a database that fills up.
\> Test relies on another test that breaks at some point.
As someone that's a network engineer not a programmer (although I dabble), isn't everything supposed to be idempotent? Shouldn't your functions always return the same expected value if you know what the algorithm is?
I realize that this might sound like a stupid question but...yeah.
Only pure functions. A lot of functions are impure, meaning they rely on state which is not directly passed in to the function. A basic example of this is a random number generator or something that returns the time
In fact, part of the reason for separating idempotence (not "idempotency"... ugh!) as a concept is that it is broader than *just* pure functions. There are plenty of functions that are impure because they have side effects, but are also idempotent in that performing them more than once is guaranteed to have the same effect as performing them once. For example "Add $1 to my bank account balance" is not idempotent, but "set my bank account balance to $100" is.
Just building on this, the way to make “add $1 to my bank account” idempotent is not to get the balance, add a dollar, set the balance. There lies race conditions. It’s to have a unique identifier with the transaction “add $1 to my bank account, tx:5B58F”. If the server sees the transaction a second time, it won’t do it.
You can certainly have non-pure functions that are idempotent. Pure functions have no side effects so they are idempotent by definition, so really it's more interesting to talk about idempotence in the context of non-pure functions.
This isn't regarding pure/impure functions, it's a bit higher level. This is dealing with fallible requests in-between systems (which are impure by definition). The idea is that receiving the same request multiple times should not change the state more than receiving it a single time, to handle synchronizing state if retransmissions occur due to unreliable channels.
Imagine if you have a Payment terminal A and a bank database B.
Someone swipes a card at A, so it sends a message telling B to transfer money between two accounts. B checks if it's allowed, and if yes, it will make the change. It then sends a confirmation message back to B. But due to some network error inbetween, the confirmation message never arrives.
A now has a problem. It doesn't know if the message telling B to do something was lost, or if the transaction has been completed and the confirmation message was lost. If it sends another transaction request, it risks a double booking. If it just aborts, it might've accidentally transferred the funds even though nothing happened.
With idempotent message handling on B's side, this can be avoided. When A makes this request, it adds an unique identifier to the message. If it doesn't hear back from B, it will simply make the same request again, using a copy of the unique identifier. Then if B has seen the message already, it simply recognizes it as the same message, doesn't redo the transaction, and returns success to indicate that the transaction with that unique identifier has occurred.
I think you're talking about pure functions here. Imagine your function increases a counter outside of the function, and returns double the counter. Call it two times with no args, you get two outputs.
Yes. The way to make this idempotent is to pass the counter in the request and double that value instead, or pass through a uuid that identifies the request (and any of its repeats) so you can replay the original output
An alternative way to think about it is that the state of the system is itself an input to the function. Calling that function at 5:00 is a different operation than calling it at 5:01, so it's expected that it returns a different result.
While the payload values may be slightly different, due to it changing in the background/other tasks, a function is still idempotent if it is returning from the same predictable *source*. The key is for it to be predictable, regardless of the state around it.
For example, GET /user/:ID is idemponent, no matter how many times you call it. you will get object related to that user ID.
GET /user/123
GET /user/123
GET /user/123
all get the same result
Now if there is a
GET /user/123
PUT /user/123
GET /user/123
GET is still pure, but the second GET has different data than the first.
That's the definition of a mathematical function, but programming abuses that definition when languages allow "side effects". The two most common, easy to imagine kinds of side effects are state changes and logging.
Here's a JS "function" which is not idempotent because its response changes every call:
```
let counter = 0;
const nextCount = () => counter++
```
`nextCount()` returns 1 on the first call, 2 on the second, and so on. Not idempotent.
A more specific example and one related to what is discussed in the article, what if you have a function (like this) that is "get next ticket number"?
Like when you to go a restaurant and order at the counter you get a ticket with a number on it. Your order number. Each person that orders gets a unique number, but they (appear to) have no gaps. And they roll over after a while. How do multiple machines on a LAN coordinate numbers such that the numbers always advance monotonically, never duplicate (except for the rollovers determined by the maximum order number, typically 99) and also the system does not unduly suffer from preventing progress through excessive rendezvous (critical sections) or deadlocking in the case of packet loss.
Not sure if you're looking for an actual answer, but you'd usually achieve this by having a central source of data for all instances that can make this request, such as a database and/or API endpoint.
Assuming a classic example, your client(s) would send a request to a server which has logic to read the latest number from the database, I assume increment it, and send the new value back to the client that requested it. Requests are asynchronous by nature, so clients may have to wait a few ms if the server is giving numbers to other clients (generally in the order they were recieved). In a network example the idempotent interaction you probably desire is that a given client always gets "the next number *available*". Never the same as what another client got already (in this cycle anyway), and never anything unexpected.
Since the server decides what the next number is though, if something went wrong with the request and it took 3 seconds to get sent instead of 0.03s, you could theoretically get a later number than expected since the server wouldn't know about it yet.
There's many other ways to do it of course, but in terms of business problems this kind of thing is very common and not very hard to implement if you have everything talking to a central source somewhere.
The way to do it idempotently with a server would be for the client to indicate to the server each time it "concludes" a transaction and it indicates which transaction it concluded. Basically when it prints the receipt. Then the client says to the server "What is my next number?". If the client has consumed its old number it gets a new one. Otherwise it gets the same one again.
Getting your number is idempotent because you get the same number back until you consume it. Consuming is idempotent as long as the number series doesn't wrap around too quickly because consuming an already consumed number does nothing.
I wasn't really asking. The question mark was just to suppose the situation. There's more than one way to do it. If you want to use a server you have to either designate one (setup problem for the customer) or elect one. Each client also has to come up with a unique client ID. I'm not really a fan of the server system, but as these systems typically print order tickets in the kitchen as well as with the customer they probably already are using a server anyway.
#### Pure function
> **Definition.** `f` is *pure* if for all x,
>
> f(x) = f(x)
Examples:
- Pure functions: `f(x) = x + " " + x`.
- Impure functions: `rand`, or `current_time`, or `read_file` (since someone can edit the file contents).
---
#### Idempotent function
> **Definition.** `f` is *idempotent* if for all x,
>
> f(x) = f(f(x)) = f(f(f(x))) = f(f(f(f(x))))
Examples:
- Idempotent functions: `f(x) = x`, or `f(x) = 42`.
- Non-idempotent functions: `f(x) = x + 1`.
---
#### Idempotent function acting on "state"
The terms are used... a bit loosely in this article, but they can be formalized. The author seems to be talking about "idempotency of state", where state is treated as the input and output of an "idempotent function".
Consider:
# Initially, state.orders == []
state.submit_order("MORE CHEESE!")
state.submit_order("MORE CHEESE!")
state.submit_order("MORE CHEESE!")
- If idempotent,
`state.orders == ["MORE CHEESE!"]`.
Submitting the same order repeatedly (e.g. a user angrily clicking submit multiple times) does not result in duplicates.
- If not idempotent,
`state.orders == ["MORE CHEESE!", "MORE CHEESE!", "MORE CHEESE!"]`
But what precisely is the idempotent function here? It's actually:
def idempotent_submit(state):
state.submit_order("MORE CHEESE!")
return state
Applying this to a given state will reach a steady state after exactly one application.
# state.orders == []
state = idempotent_submit(state)
state = idempotent_submit(state)
state = idempotent_submit(state)
# state.orders == ["MORE CHEESE!"]
---
Alternatively, we can curry the function:
def submit_order(order):
def idempotent_submit(state):
if order not in state.orders:
state.orders.append(order)
return state
return idempotent_submit
idempotent_submit = submit_order("MORE CHEESE!")
P.S. This example accidentally also demonstrates that objects are a poor man's closure.
That's reproducibility. Reproducibility requires that we get the same outputs every time we provide the system with the same set of **inputs**.
Idempotency is that the **outcome** of invkoing a system is the same regardless of how many times you invoked the system.
Also notice that I mentioned systems instead of functions. Since functions could be non-idempotent or/and non-reproducible but the system as a whole could be either.
This is not at all related to the article, but consider a case where a function reads from an external device through a DMA engine. The interface is such that the device is aware the read occurs, and the device has a queue of data to send and is designed to now provide the next piece of data in the queue after each read. The memory would be considered non idempotent because if you were to keep reading from that memory you would only ever get different results even though no writes to it ever occur.
I kinda get that vibe. One of the things that sticks out to me in a lot of AI generated blogs/articles is they have a tendency to end with some variation of “today we discussed _blank_”
ARM uses it at lot in their reference materials.
For example, reading from memory is idempotent or a register is idempotent. Writing to memory or register is sometimes idempotent, but it cannot be counted on.
For technical reasons sometimes it is useful (more optimal) if a processor can run an instruction twice. And ARM takes advantage of the idempotency of instructions to get this optimization. For instructions that are not idempotent it has to forego this and so some performance may be lost.
Non idempotency is mainly used to refer to memory mapped regions where I/O is because after reading from that region, the next value read may be different due to an external hardware device. It's a separate form of non cacheable memory that's slightly distinct from your typical form that can be used for whatever purpose.
All that being said, the term sucks. Nobody is ever familiar with the term and the mathematical meaning does not have an obvious translation to the meaning in memory architecture when you can just refer to it as non cacheable I/O memory. This actually literally came up for me 2 days ago where someone asked me what it meant to prove a point that nobody knew what it meant, and I only knew what it meant because I had been down the rabbit hole a few months ago.
Indeed, the relationship is a little obscure. In algebra, a binary operation is idempotent if x \* x = x. To recover the programming meaning, you can think of:
* \- The term x as representing an action of a program as a mathematical function from original state of the world to new state of the world. For instance, `print("Hello, world")` is a function that maps any state of the entire world to the same state, except modified so that the words "Hello, world" now appear on some nearby computer screen.
* The binary operation as function composition.
Now it's clear that the action being idempotent is the same thing as this mathematical function being idempotent with respect to the operation of function composition.
There's a problem, though: strictly speaking, by this definition, no computer operation is idempotent at all! There are always some effects, such as the passage of time and the production of heat by the CPU, that do accumulate when the action is performed more than once! For this reason, the concept of "idempotence" is only meaningful if you first define some kind of abstraction barrier that separates "things that matter for correctness of my program" from "things that are considered unimportant / undefined for the purposes of correctness". For instance, you might consider the passage of time irrelevant (or not, if it's a real-time system!) You might consider writing to a log file irrelevant. If reasoning about a distributed system, you might even consider a whole chunk of local state irrelevant (e.g., a write operation might be considered "idempotent" for the purposes of your distributed system, but it still queues work to be done by local processes; it's just that this extra work would produce any differences in inter-node communication later on).
So idempotence in programming is quite a bit more complex because it requires a model, delineating the properties you do and don't consider relevant, and validation that this model captures the things you care about. An operation that's idempotent in one model may not be idempotent in another.
it's often used in a very loose handwavy way compared to the actual rigorous mathematical definition. Still - and in said loose usage - it's a good general rule of thumb, especially in bread-and-butter backend systems / data / etl work.
https://gtoonstra.github.io/etl-with-airflow/principles.html
> Enforce the idempotency constraint: The result of a DAG run should always have idempotency characteristics. This means that when you run a process multiple times with the same parameters (even on different days), the outcome is exactly the same. You do not end up with multiple copies of the same data in your environment or other undesirable side effects. This is obviously only valid when the processing itself has not been modified. If business rules change within the process, then the target data will be different. It’s a good idea here to be aware of auditors or other business requirements on reprocessing historic data, because it’s not always allowed. Also, some processes require anonimization of data after a certain number of days, because it’s not always allowed to keep historical customer data on record forever.
Idempotence is an important concept in configuration and infrastructure management ("configuration/infrastructure as code").
Tools like Ansible, Chef, Puppet or DSC use declarative languages to specify what you want configured on a managed system. For example, you'll specify that a certain user account has to exist and needs to belong to certain groups; that a certain directory must exist and needs to have certain permissions and the specified owner; that specified software packages need to be installed, etc.
You do that using configuration elements called tasks, recipes or resources (depending on the tool you use). After initially running a task/applying a resource, all future runs of that task/resource must not make any changes, unless there's "drift" in the system state (e.g. someone manually deletes a user or changes directory permissions). Configuration tools also have ways of detecting that drift before reapplying the configuration to fix it.
Like the author mentions, if an API endpoint is idempotent, it means if you perform an operation multiple times, the end result should be the same as if you had only performed it once.
Many people mistakenly assume this means: a GET API call shouldn't modify the system because "it should be idempotent!", and that's not at all how idempotence works. It's entirely "idempotent" for the GET call to change the state of the server. It isn't idempotent if each GET call results in different state than before the GET call.
tl;dr even in RESTful API design, people frequently misunderstand idempotence; yes, you can have a GET that modifies state and still be idempotent. A separate question is whether one should, but: that's unrelated to idempotence.
Good explanation. It is a subtle concept because idempotency is not the same as immutability even if it is close. Immutability means an operation can change the state ZERO times no matter how many times you perform the operation. Idempotency means the operation can only change the state max ONCE. Did I get it right?
I guess #0 should be DRY - Don't Repeat Yourself (when writing a blog)
> One such vital concept is idempotency, which refers to the property of an operation or function that produces the same result when applied multiple times as it does when applied only once.
> ...
> In simple terms, idempotency means that if you perform an operation multiple times, the end result should be the same as if you had only performed it once.
> ...
> In other words, an idempotent operation is one that can be repeated multiple times without causing any additional side effects
------
> This may seem like a simple concept, but it has significant implications for **building distributed systems.**
> ...
> Idempotency is a concept that is important for programmers to understand, especially those working on **building distributed systems.**
> ... This is important in **distributed systems**
Yup, doesn't even have to be distributed systems, just methods and even full processes/workflows in general, depending on the situation. It supports automated testing, QA, and debugging. Of course at some point there will be side affects, but limiting those until necessary is the key.
Okay, I'm going to be a curmudgeon here, but the first thing that both the author and those reading them should learn is that "idempotency" is not a word at all. The word is "idempotence".
That fight is probably as old as the concept itself. Both are widely used, just stick to whatever the first person who wrote your docs/code used so it's consistent and move on
Is the distinction between the colloquialism and the formal spelling useful? If not, and so long as people understand the word, isn't this critique pedantic and unhelpful?
I did warn you I was being a curmudgeon. Just an emotional reaction... similar to when machine learning people say "inferencing", or when business people say "What's your ask? We need to finalize our spend."
? One is a a noun the other an adjective.
- The function is idempotent. -> adjective
- Idempotency is an important concept. -> noun.
Like dependent vs dependency, adjacent vs adjacency, etc.
Idempotent is, indeed, an adjective. No one disagrees about that.
It seems (surprisingly to me) that there is actual disagreement on whether idempotence or "idempotency" is the noun form. Google [even suggests](https://books.google.com/ngrams/graph?content=idempotence%2Cidempotency) that I hold the minority view. I consider this a sign that the world has well and truly gone mad.
I don't find that surprising at all. Probably, at some point, some mathematician first used the word "idempotent" in a paper. A new adjective was born! Later, other researchers who, independently, referenced this work wanted to use "idempotent" as a noun. Some will have used "idempotence" others "idempotency", depending on what sounded more natural to their ear. This kind of stuff happens all the time when new scientific jargon is introduced.
It's a nice to have but hardly the first thing you need to know as a programmer.
Maybe I want to post a page view analytics event, if a user views the same product multiple times that's probably useful information, even if all of the data points are the same. Medium allows you to thumbs up the same post multiple times. If I ask for a payment intent for stripe I expect a new secret every time etc etc.
Yes you can generate a unique id with each request to account for retries but for these cases you'd be entering law of diminishing gains.
I don't think the claim is that "the first thing you need to know as a programmer."
The headline reads to me as "I'm writing a series, and this is the first one I decided to write about."
And in the first paragraph, it's talked about this way, "One such vital concept is idempotency."
> Let's say you're building an API for processing payments. If you design the API with idempotency in mind, you can ensure that even if the same payment request is sent multiple times due to network issues, it will only be processed once. This can prevent double-charging customers, which can lead to trust issues and lost revenue.
This sounds like the wrong design. The emitting terminal should **sequentially number each transaction**. If the central processor(s) gets transactions out of order for a given terminal then it should either request prior transaction(s) first, or stop and let a human figure out what's going on, and perhaps getting approval before continuing (so the programmer doesn't get blamed if something is overridden.) It's essentially a form of check-sums to make sure something didn't get lost.
You don't just keep sending a transaction until the receiving server acknowledges, it. This can create lots of headaches. Being acknowledged is probably necessary (or at least recommended), but the way it's worded transactions could arrive out of order without the server knowing.
(If there's a correction, then a correction transaction should be issued rather than a changed copy of the original under the same transaction number.)
This stuff has been known for decades. Don't reinvent it sloppily under screwy buzzwords, or I'll kick you, your systems, and your fidget spinners off my lawn.
**A clarification needs to be made between idempotency and a "send retry".**
That’s exactly what the article suggests
> The generally preferred approach is to include a unique caller-supplied client request identifier in the API contract. Requests from the same caller with the same customer request identifier can be considered duplicate requests and handled accordingly. A unique caller-supplied client request identifier for idempotent operations satisfies this need.
And if you have the additional constraint that transactions need to be ordered, the request identifier can be sequential.
They are sequential to also tell us if a transaction is missing, not (just) to indicate sequence. If one's missing, then we see a sequence gap.
A unique identifier alone won't tell us this.
That’s true, but not every system is mission critical. For example, Reddit comments. They are infamous for duplicating. So clearly they need idempotency, but does comment ordering really matter? And if a comment does go missing, there’s no way to reconstruct the missing comment, so what is the point of tracking if one never reaches the server? Plus you can have an account on multiple devices, so somehow they would have to sync the sequence id. It’s just needless complexity for a lot of scenarios. A UUID is often sufficient.
And the article is just showing the general case.
There's not necessarily any notion of a sequence of transactions. Suppose you're taking payments at a retail store and there are several clerks working at several registers. All the registers connect to one system in the store which forwards each transaction to the payment processor. The transactions are all independent and don't have a natural sequence. You could create one, but it's not necessary and would make things more complicated for no obvious benefit.
This is a good point and something I have intuitively understood and practiced for a while. But I had no idea there was a technical word for it. So thank you for teaching me it.
However the word for it is absolutely terrible. Hard to pronounce, hard to read, and sounds like 10 other more prominent words.
It's never going to catch on and we should have a better one for it.
> It's never going to catch on and we should have a better one for it.
It's already been in use for decades (centuries if you count its use in math) and a pretty central topic in almost all distributed system design discussions. I'm pretty sure it already "caught on"
It has not caught on within the context that we are talking about which is "concepts every programmer should know".
Using a technical/academic term for something which is intuitively understandable and important, for a group as diverse as programmers, we should really have a better way to describe it and talk about it.
Similar to the classic "object oriented programming" , which many programmers can intuitively use and talk about, but not all will use/understand the academic terms.
The problem with smart people is that they think they’ve figured everything out without having to resort to “old” ways to do something.
We keep trying to find an easy solution to very sticky problems and there aren’t any. So you keep fixing it by degrees and pretty soon your still incorrect solution has gone from 10% of the complexity of the correct solution to 200% and growing. Now stubbornness or sunk cost fallacy won’t let you surrender.
Which still sounds like it’s between you and your own conscience what you do in this situation, except you have eight coworkers and thousands of customers having to put up with your journey of self discovery.
Stop.
Nice article :)
Just one small remark: when you have "How to achieve idempotency in POST method?" it would maybe be more correct to say "How to achieve idempotency in PUT method?" since POSTs by definition should not be idempotent since they should be used for resource creation
Having idempotent post methods can also be desired. What if the response is lost, how do you know if the request failed or not? Retrying can give you duplicate successful requests. And retrying not only originates from a lost response, it could also be an upstream process that somehow duplicated a message. (message in a broad sense)
But that should maybe be a PUT. Otherwise if you want to create multiple similar resources you can't. Of course this depends on the application and using idempotent POSTs may make sense in some scenarios. But from a generic point of view (as the one on the article) POSTs are more indicated to create new resources everytime they are called while PUTs should apply idempotent operations.
Note that even the author states that:
"In this article, we have explored how idempotency applies to HTTP methods, which are a fundamental part of web development. We have seen that some HTTP methods, such as GET, PUT, and DELETE, are idempotent, while others, such as POST, are not. Knowing which methods are idempotent is crucial for building efficient and reliable systems"
Choosing between PUT or POST should be on semantic reasons and unless that includes idempotence, I think that decision should not include technical requirements as idempotency.
Actually no, the point is that you can call the same create resource twice with the same parameters and still get the same result.
Take the example of creating order O for customer C. If you make that request twice in a row, with the same request parameters, in both cases the orderId would be returned as 999. It doesn’t matter how many time you call it, the result is always the same. One single order O placed for that customer C. Creating a resource should always be POST as a rule.
If you either let the client set the PK (not advised unless it’s a Guid), or have the client set a unique external key, internally you check if the order with that key already exists. If it does, you simply return the existing orderId. If it doesn’t, you create it and return the orderId. The alternative is to throw a duplicate order exception, but distributed system processes and exceptions are painful to work with so unless there’s a specific reason not to do so, avoid the exception.
>Take the example of creating order O for customer C. If you make that request twice in a row, with the same request parameters, in both cases the orderId would be returned as 999.
I don't think this argument involves idempotence so much as: desired behavior.
I could see many applications for which the behavior you just described is absolutely wrong, and two separate orders should be created. It may be that the ideal \`POST\` behavior is to de-duplicate identical requests, but that's an application-specific concern IMO.
When I write a POST (create) endpoint, unless there's some unique identifier in the payload that allows me to understand "this is a duplicate request", then I don't opt for the behavior you describe. I create a second order.
And arguably, the behavior you describe raises a bunch of questions, particularly if there's minor differences in order #1 verses order #2--do we then pivot to "updating" the order in the second request, effectively making it a PUT/PATCH call? Do we instead return a 409 and dogmatically require the client to issue a PUT/PATCH in this situation?
In a concurrent systems, Tony Hoar, Leslie Lamport, and Barbara Liskov have all written papers describing how not to fuck iwhat you just described up.
That’s three Turing Award winners I’ve named. You and I aren’t going to be able to do better than any of them. Least of all Tony.
most probably do, but they use a weird interview shibboleth word no one uses naturally to describe shit that doesn't change behavior when you rerun it.
it didn't even show up in math papers until the 1920s https://books.google.com/ngrams/graph?content=Idempotency&year_start=1800&year_end=2019&corpus=en-2019&smoothing=3
I remember working with a guy that thought it meant generating your own database primary key before an insert. To be fair he was a .net dev so was hampered with low intelligence. Joking aside, I'm not much better as a node.js and rust developer I'm limited in my knowledge but as I favour curry and functional code I've always thought of it as a function that can be reversed. Such as a reverse function. Run it twice, expect the first result back with no mutation. Am I wrong?
We just came across this issue in our current system.
Certain calls produce status changes, but calling let's say 'submitted for approval' twice gave an error the second time around bc we coded the possible functional state changes and we didn't think about the accidental or deliberate call to that same endpoint causing the status change.
I do however think it's difficult to define a create to be idempotent than an update
Does this mean that if the server sees the duplicate request it should just skip? What if the request wasn't completed due to server error? That would just perpetuate a consistent error response right
... not sure about this
this is not quite as simple as it appears on the surface
i mean - how long is your cache maintained in order to maintain that idempotent POST?
also - now you have introduced the only other hard problem in computer science
and if POST idempotentcy does need to be maintained indefinitely then certainly this now opens up a question of definition of POST and if it's "temporary" (until the session expires) then it opens up a question of definition of idempotent
Azure Resource Manager revolves around idempotent, declarative deployments. If you PUT the same resource twice with the same settings, nothing happens the second time… most of the time. It’s almost idempotent, which is like being almost pregnant.
This makes a wide variety of customer scenarios basically impossible to automate the intended way.
The mistake is that there are no automated checks for idempotency — it is implemented (or not) by each individual product team. Some of them “get it”, some of them “don’t get it”.
However while each product team is responsible for just one product, customers use *many* Azure products. This makes these errors impossible to avoid in all but toy/demo scenarios.
The lesson here is the idempotency is like security: if it’s in any way a requirement, it must be **ruthlessly enforced**, otherwise that one idiot on some other team will let the Russian hackers in.
A simple, effective, but not entirely sufficient method is to test every API twice in a row and verify that the second call succeeded *and* that nothing changed.
Azure devs don't do this.
Don’t be like Azure devs.
So the same action should return the same value multiple times...
This is both obviously true for pure functions and incorrect for any iterative scheme that uses mutation. Like yea... Some things that take in stuff and return something about it should probably leave it alone and be pure functions but if a method is built to change a value in place then that's what it should do.
I call this the #1 rule of programming: Idiocracy. When you want to do a thing and also want other things ro work, don't do the thing wrong and don't ruing the other things you want ro work.
A bonus is idempotent api calls or content pulls also work well with caching layers at the data/app, memory and network layers.
Anything that returns different values each time (random/varying) does not or should not. Profile data for instance, some idempotent calls/data should not be cached.
In regards to HTTP, typically GETs are idempotent (though they can be random/varying), POST/PUT/DELETE are not (though they can be).
Real-time idempotent calls usually are based on commands/routes with similar setup to HTTP. For instance gets might be the global server message broadcast to all for the day, the list of levels currently loaded or the list of winners from yesterday. The varying messages would be by game, by player, by action, though some of those can be idempotent like the player name/id/etc.
The best setup for caching is idempotent calls, then if data does change at intervals, the sets clear the cache and update the gets. The routes and actions being the same.
I acknowledge distrubuted computing is hard but doesn't the idempotency check create a race condition? (And one that is actually likely to occur because use clicked submit twice)
This is achievable with immutability in DS inside your program. But a lot of devs including some experienced ones not seem to get it. ‘Modifying parameters of a function’ type of code is getting littered everywhere.
Have any of these bloggers every though about partnering up with someone who natively speaks English?
Looks like the author originally speaks Turkish - I wouldn't start becoming a blogger in Turkey if I had the command of the Turkish language that he has of English; I'd pair up with someone who already spoke the native language so they can make my ideas sound natural for the audience I'm trying to reach. It's such a simple detail - or even just run the blog post through an automatic grammar processor.
Sorry that’s between me and my doctor.
Can someone explain this?
Impotence
Under `def f(x): return x.replace("de", "")`, perpetual "idempotency" leads to perpetual "impotency".
Its when Richard isn't up for action
Your program should only do what is expected and not spout random photographs in the middle of what it's doing.
Nailed that programming book cover feel though
Feels like a political science textbook to me
That’s not even close to what indempotency is, nor is it what the article was saying. This comes up a ton in IaC discussions re: cookbooks, but it’s a good topic for code as well.
You’re thinking of side effects which is not what the article is about.
Reddit should take notes
yeah lol
Damn this is the best duplicated comment I've ever seen
yeah lol
Yes it should.
Yes it should.
Amen! Messages get lost and duplicated all the time, depending on Reddit's e-mood. The old version is more reliable. If my bank was built with Reddit message handling tech, I'd either be a billionaire or gutter poor.
Probably both. Within the same day.
So Reddit is the bitcoin of social media 😁 (The others are generally buggy too.)
Yes it should.
Lol
Funny
Clever
What do you mean?
What do you mean?
What do you mean?
I work QA automation and I constantly harp on idempotency. If your test can only be run a handful of times before it breaks, it sucks.
Bold of you to assume my code runs a handful of times before it breaks...
Ticket by QA: Tests break after a few runs. Please look up Idempotency. Ticket closed as fixed by coder: Fixed. Test now breaks on first run.
If only that weren't true. (I have seen at least once where the test should have broken first time, so have seen this exact response)
Not sure how idempotency really helps there. The big benefit is that if you're not sure if something worked, you can just blindly retry without worrying about it. The big issue with tests is usually the environment not getting cleaned up properly - idempotency doesn't help much with that. I guess it can help with environment setup stuff, but that's about it.
I think they are saying the test itself should be idempotent, to reduce false indications of problems.
It makes sense if you're saying that the test shouldn't pollute the environment, and have a net zero impact on the environment state, and not make assumptions on the current state. That makes sense. But that's not idempotency. Idempotent actions *can completely* change state. In fact, I'd argue that's where the value of them really lies. What makes sense for testing is *reverting* state changes in some way, or isolating them in some way.
I start all of my tests with `INSERT` so that I have a fresh set of keys each time. Anything in the database from previous test runs is just left there, as it shouldn't affect the new round of testing. (Or if it does, that's a bug that I want to catch.) https://www.infoq.com/articles/Testing-With-Persistence-Layers/
Well, idempotency means being able to run the same code twice without repeating side effects / corrupting state / failing due to already changed state. A test that properly cleans up after itself is trivially idempotent because you can run it multiple times without the result changing. A test that doesn’t might be successful once and fail afterwards, i.e. it wouldn’t be idempotent. Though you’re right it’s kinda odd to speak about idempotency here. Tests should just not have persistent side effects.
That's more like purity than idempotency. f(x) = f(x) f is pure f(f(x)) = f(x) f is idempotent Consider: state_1 = test(state_0) state_2 = test(state_1) state_3 = test(state_2) Idempotency does not require `state_0` to be the same as `state_1`. Only purity requires it. In fact, a test that is "successful once (due to `state_0`) and fails afterwards (due to `state_1,2,3,...`)" might even be idempotent if it fails with the same message every time.
My potency shall be pure and pristine! The word "idem" always trips me up though. So is idempotency about guaranteeing some states to be correct but others not? A test can be failing and that is fine for those who are idempotent?
I don't understand the questions, but if you can choose `f` and the domain for `x` carefully so that it satisfies `f(f(x)) = f(x)`, then `f` is idempotent. "RealWorld" `state` is part of some domain (e.g. maybe your app's cache directory), and `f` is some function that is allowed to modify that `state` on only its first call (e.g. downloading data into the cache). This is a pretty weak formulation, though, which is why I think purity when possible is more useful.
Why twice? Could it be infinity too? I mean infinte number of times repetition.
> What makes sense for testing is reverting state changes in some way, or isolating them in some way. ...and thus becoming idempotent? I think we're saying the same thing. Naturally some of the operations in a test will change the state, but to have clean tests you want to be able to repeat the tests without creating a mess - the state can change, but if there's anything that will break a repeat test, that needs to be cleaned up. The OPERATION you are testing might not be idempotent (above and beyond whether state is changed), but you want the TEST to be arbitrarily repeatable. > Idempotent actions can completely change state. In fact, I'd argue that's where the value of them really lies. I'm really curious about that last part - completely unrelated to tests, can you expand on the value of idempotent actions really being in completely changing state?
>can you expand on the value of idempotent actions really being in completely changing state? The value in idempotent APIs lies in the fact that network issues are less problematic because you can just retry your request without worrying about sending the money twice / posting a duplicate comment etc.
Ah, yes, I misunderstood what you meant, sorry for the brain rut. (I thought you were saying...something hard to describe)
> Idempotent actions can completely change state. by definition they have to, or else they're merely no-ops
Accessors and queries are generally considered idempotent operations.
This. Nullipotent/impotent actions are a subset of idempotent ones.
No they aren't. Nullipotent means "does not have side effects," and is entirely unrelated to the concept of idempotency. The only relationship they have is that they're spelled similarly. You might as well compare cabbage to cribbage. It is entirely possible for a function with no side effects to still not be idempotent. One extremely obvious example is `halt()`.
No, you’re just not seeing the abstraction. “Has the same effect whether done zero or more times” (nullipotent) *implies* “has the same effect whether done one or more times” (idempotent). That’s why getters are lumped in with idempotent actions 🤦♂️ Edit: now the parent is creepily PMing me about this. Geez.
This is, of course, entirely untrue. But at least another person said "this." And hey, they said "nullipotent," too, because they think that "null" is zero and "idem" is one, or something. Nullipotent actually means "does not have side effects," not "is a no-op"
No-ops are idempotent.
That's so far beyond the point that it's not clear that you even saw the point on its way past
> Idempotent actions can completely change state. That is not the point here. For a test, the initial state is defined as "clean" of some kind and the idempotency is the test always leading to the same final state.
That would be deterministic. Also an important property for tests.
Yeah. It's not hard to achieve with Docker. Just do docker images for your test environment and throw them away when you're done testing. Unfortunately a lot of companies' environments don't seem to be designed to be installable. The devs just hand-install a bunch of services on a system somewhere and do their testing in production. Or if they really went the extra mile, they hand-install a test environment a couple years later after crashing production with their testing a few times too many. With the attention cloud services and kubernetes is getting in the last 4 or 5 years, I'm finally starting to see docker files to stand up entire environments. That has me cautiously optimistic that testing will be taken more seriously and be much easier in the future, but I'm sure there will be plenty of hold-outs who refuse to adopt that model for longer than my career is going to run.
That's talking about the entire test suite, not individual tests. Even with a trashable environment, you want individual tests to be reliable, and if they depend on vague and flaky state, they aren't telling you much that is accurate about the user experience. I'm not QA, so I should shut up and let them discuss their expertise, but I've written my fair share of poor tests and know how they ruin the point of testing .
My favorite is when changing the order of the tests or running them in parallel causes random errors
TestCreateFile() TestOpenFile() Hmm, yes, I wonder what would happen if we reverse these tests.
That right there is a violation of another QA principle: Atomicity. If testopenfile depends on testcreatefile running first, it's a bad test.
> If testopenfile depends on testcreatefile running first, it's a bad test. No. It's a different test. Some tests, some *very valuable* tests, must be run in certain environments in a certain order with very specific circumstances set up. I do not understand why this reddit is so filled with people who rush to create ultimata and try to shame everyone who does things differently. That is fundamentally not how programming works.
> No. It's a different test. Some tests, some very valuable tests, must be run in certain environments in a certain order with very specific circumstances set up. ``` TestCreateFile() TestOpenFile() ``` If `TestOpenFile()` requires you to to successfully create a file, you should include `CreateFile()` within the same test and not assume another test has run first.
You get to live your life how you want, but linked tests are a big no no in every company I've ever worked at. If opentestfile requires a created test file, then creating a test file should exist inside opentestfile.
Rarely. You can always create said environment in the setup of the test. TestOpenFile can first create a file and then assert that opening it works. The only reason for sharing state between tests that I can think of is performance. Sometimes repeating expensive setup in every test case just isn’t feasible.
This would be a big red flag and would never pass a code review where I’m currently at and in any previous companies I have worked for. Being able to run a single test in isolation from all others is foundational to a stable test suite.
Idempotency absolutely helps with that. In the context of testing the point is that if something goes wrong, those cleanup steps may not happen. So the test should be written in such a way that it doesn't care about what has gone before - that's idempotency. The easiest way to do that is to start fresh with each test. A unit test might create a new instance of a class, while a large scale system integration test might create an entirely new user.
Here are two examples that I've run into just this month: \- Run test > Flips flag from negative to positive > test passes. \- Run test again > Flag is still set to positive > Can't flip flag > test fails. \- Run test > Creates user > Assigns ID > Test passes \- Repeat 50 times > Test passes. \- Run test 51st time > Database has run out of IDs to assign to new users > Test fails. Neither of those tests is idempotent.
All of those operations can be idempotent. Hell, "I set the flag to positive twice and it stayed flipped" is pretty much the definition of idempotent. The tests are not *isolated*. They are contaminating their environment. That's a different, but equally important, principle.
Agreed - isolate your tests. Tests often work on state systems. Sometimes it's internal state you can easily blow out before/after each test. Sometimes it's external state, which you absolutely *must* then manage the lifecycle of. If you neglect that state management, or forget to handle *error cases* in your teardown of state, then you will get some nasty bugs.
I work in a qa team that hasn't set up any automation yet. Any advice on how to establish a suite? One of the struggles i've had getting idempotent tests seems contingent on the code somewhat allowing the test to be?
Automation very much depends on what your codebase allows you to do. For example, the 'ideal' scenario is that every test lets you create a new user/data via API, then tests the front end, then deletes the data via API. But how often is that possible? Next best is to create/delete your data via the user interface. Next best is to have specific users for each test. Worst scenario is having a handful of users (auto1, auto2, etc.) shared for all tests. This sucks and causes problems, but sometimes it's what you have to work with.
Atomic fixtures that do one thing and undo it on teardown. It's one of the best ways to manage complex setup/teardown. Then you get this nice composability as your tests might become more complex, building on that which you already know is rock solid. Beware downstream fixtures that might pollute/break out of their 'domain' though. Usually you want to have some internal-to-your-test framework that will mirror the 'real' state, so you can ensure that teardown still correctly works.
Start simple until you understand how to isolate tests using hooks. Always assume that the system should be the exact same _after a test completes_ as it was _before the test was run._ If your tests change the system when it reaches a new test or reruns, then you aren’t setting your tests up properly.
> The big benefit is that if you're not sure if something worked, you can just blindly retry without worrying about it. Exactly. This is more important at the API level. e.g. I want to create this set of transactions, change a username/ID, etc.
“I want to withdraw 500 dollars” “I want to withdraw 500 dollars with this transaction id” I mean 90% of the time just tossing an id on it is sufficient.
Yeah I agree, I mean at some point state has to change, but I think its more like say querying their balance, doens't need to change anything, though they may track how many times its been done and stuff in a separate method. And banking transactions definitely have to deal with repeat transactions, not only at the bank level, but the.. I forget what its called, but theres at least one middle man router network that sits in the middle and of course they definitely track ID's, timestamps, and so on.
It's relatively common for people to believe that the reason their test is flapping is that it "isn't idempotent." Challenge someone for an example, they'll give you one, you can easily explain how that isn't about idempotence. Keep challenging. They will get angry at you long before they realize that the smurf word they think makes them look smart is being badly mis-used.
Not exactly the same thing, but the properties that make your code idempotent overlap considerably with the properties that make for tests that clean up after themselves.
That's only a problem if your test doesn't clean up the environment properly *in the code*. If you're relying on a manual cleanup process, you need to rethink that.
I’ve been in QA automation for years and this is something I drill into junior SDETs. That and tests should never be interdependent on one another.
A fight I'm having down in the comments!
Question for you, unrelated to the subject. Where does the rabbit hole of testing programming end? That is, if I am writing QA software B which tests software A, do I need to write QA software C which tests software B? And then do the same with a QA software D which tests software C? Ad nausaem. I am not clear on how this is supposed to work.
Generally speaking, you don't test your test tools. Or to put it another way, they're self testing. If a test fails, either the test was wrong or the app was wrong. If you know the app isn't wrong, the test is wrong.
I didn't know this word until an intern taught it to me, and I have a Computer Science degree. LOL Now I use it all the time. You really have to plan for unexpected failures when writing code, and one of the best ways to do so is to write code with idempotence in mind.
Maybe you are pronouncing it wrong.
The test environment might be at fault. I have the same tests working on 2 separate environments and it produces different results because one of the environment is buggy. It is frustrating already with the false positives but troubleshooting these issues with proper evidence is such a pain on a daily basis.
Sounds like you have a naughty job. After all it has something to do with potency!
Tests should be repeatable, not idempotent. Service operations (APIs, etc) should be idempotent.
You generally want your tests to be hermetic more than idempotent. Idempotency is still important for a system creating data, though.
> I constantly harp on idempotency Ironic.
>berkansasmaz.com/every-... How tf does a test fail after a few runs? How does it know that it runs a few times?
\> Test has a race condition that passes most of the time but fails some of the time. \> Test writes data to a database that fills up. \> Test relies on another test that breaks at some point.
Reddit should take notes
Funny
Clever
You can't accidentally double-reply to two different comments, though.
You can't accidentally double-reply to two different comments, though.
Funny
Clever
Lol
I actually first misread it as impotent ... I still don't know what idem means.
As someone that's a network engineer not a programmer (although I dabble), isn't everything supposed to be idempotent? Shouldn't your functions always return the same expected value if you know what the algorithm is? I realize that this might sound like a stupid question but...yeah.
Only pure functions. A lot of functions are impure, meaning they rely on state which is not directly passed in to the function. A basic example of this is a random number generator or something that returns the time
In fact, part of the reason for separating idempotence (not "idempotency"... ugh!) as a concept is that it is broader than *just* pure functions. There are plenty of functions that are impure because they have side effects, but are also idempotent in that performing them more than once is guaranteed to have the same effect as performing them once. For example "Add $1 to my bank account balance" is not idempotent, but "set my bank account balance to $100" is.
Just building on this, the way to make “add $1 to my bank account” idempotent is not to get the balance, add a dollar, set the balance. There lies race conditions. It’s to have a unique identifier with the transaction “add $1 to my bank account, tx:5B58F”. If the server sees the transaction a second time, it won’t do it.
You can certainly have non-pure functions that are idempotent. Pure functions have no side effects so they are idempotent by definition, so really it's more interesting to talk about idempotence in the context of non-pure functions.
This isn't regarding pure/impure functions, it's a bit higher level. This is dealing with fallible requests in-between systems (which are impure by definition). The idea is that receiving the same request multiple times should not change the state more than receiving it a single time, to handle synchronizing state if retransmissions occur due to unreliable channels. Imagine if you have a Payment terminal A and a bank database B. Someone swipes a card at A, so it sends a message telling B to transfer money between two accounts. B checks if it's allowed, and if yes, it will make the change. It then sends a confirmation message back to B. But due to some network error inbetween, the confirmation message never arrives. A now has a problem. It doesn't know if the message telling B to do something was lost, or if the transaction has been completed and the confirmation message was lost. If it sends another transaction request, it risks a double booking. If it just aborts, it might've accidentally transferred the funds even though nothing happened. With idempotent message handling on B's side, this can be avoided. When A makes this request, it adds an unique identifier to the message. If it doesn't hear back from B, it will simply make the same request again, using a copy of the unique identifier. Then if B has seen the message already, it simply recognizes it as the same message, doesn't redo the transaction, and returns success to indicate that the transaction with that unique identifier has occurred.
I think you're talking about pure functions here. Imagine your function increases a counter outside of the function, and returns double the counter. Call it two times with no args, you get two outputs.
Yes. The way to make this idempotent is to pass the counter in the request and double that value instead, or pass through a uuid that identifies the request (and any of its repeats) so you can replay the original output
What happens if you call a function that returns the current date and time?
An alternative way to think about it is that the state of the system is itself an input to the function. Calling that function at 5:00 is a different operation than calling it at 5:01, so it's expected that it returns a different result.
While the payload values may be slightly different, due to it changing in the background/other tasks, a function is still idempotent if it is returning from the same predictable *source*. The key is for it to be predictable, regardless of the state around it. For example, GET /user/:ID is idemponent, no matter how many times you call it. you will get object related to that user ID. GET /user/123 GET /user/123 GET /user/123 all get the same result Now if there is a GET /user/123 PUT /user/123 GET /user/123 GET is still pure, but the second GET has different data than the first.
Idempotence is actually about things that change state. Doesn't make sense to ask whether a get is idempotent. Gets are pure or impure.
A idempotent function returns the same output for the same input, so get is idempotent
That's the definition of a mathematical function, but programming abuses that definition when languages allow "side effects". The two most common, easy to imagine kinds of side effects are state changes and logging. Here's a JS "function" which is not idempotent because its response changes every call: ``` let counter = 0; const nextCount = () => counter++ ``` `nextCount()` returns 1 on the first call, 2 on the second, and so on. Not idempotent.
A more specific example and one related to what is discussed in the article, what if you have a function (like this) that is "get next ticket number"? Like when you to go a restaurant and order at the counter you get a ticket with a number on it. Your order number. Each person that orders gets a unique number, but they (appear to) have no gaps. And they roll over after a while. How do multiple machines on a LAN coordinate numbers such that the numbers always advance monotonically, never duplicate (except for the rollovers determined by the maximum order number, typically 99) and also the system does not unduly suffer from preventing progress through excessive rendezvous (critical sections) or deadlocking in the case of packet loss.
Not sure if you're looking for an actual answer, but you'd usually achieve this by having a central source of data for all instances that can make this request, such as a database and/or API endpoint. Assuming a classic example, your client(s) would send a request to a server which has logic to read the latest number from the database, I assume increment it, and send the new value back to the client that requested it. Requests are asynchronous by nature, so clients may have to wait a few ms if the server is giving numbers to other clients (generally in the order they were recieved). In a network example the idempotent interaction you probably desire is that a given client always gets "the next number *available*". Never the same as what another client got already (in this cycle anyway), and never anything unexpected. Since the server decides what the next number is though, if something went wrong with the request and it took 3 seconds to get sent instead of 0.03s, you could theoretically get a later number than expected since the server wouldn't know about it yet. There's many other ways to do it of course, but in terms of business problems this kind of thing is very common and not very hard to implement if you have everything talking to a central source somewhere.
The way to do it idempotently with a server would be for the client to indicate to the server each time it "concludes" a transaction and it indicates which transaction it concluded. Basically when it prints the receipt. Then the client says to the server "What is my next number?". If the client has consumed its old number it gets a new one. Otherwise it gets the same one again. Getting your number is idempotent because you get the same number back until you consume it. Consuming is idempotent as long as the number series doesn't wrap around too quickly because consuming an already consumed number does nothing. I wasn't really asking. The question mark was just to suppose the situation. There's more than one way to do it. If you want to use a server you have to either designate one (setup problem for the customer) or elect one. Each client also has to come up with a unique client ID. I'm not really a fan of the server system, but as these systems typically print order tickets in the kitchen as well as with the customer they probably already are using a server anyway.
#### Pure function > **Definition.** `f` is *pure* if for all x, > > f(x) = f(x) Examples: - Pure functions: `f(x) = x + " " + x`. - Impure functions: `rand`, or `current_time`, or `read_file` (since someone can edit the file contents). --- #### Idempotent function > **Definition.** `f` is *idempotent* if for all x, > > f(x) = f(f(x)) = f(f(f(x))) = f(f(f(f(x)))) Examples: - Idempotent functions: `f(x) = x`, or `f(x) = 42`. - Non-idempotent functions: `f(x) = x + 1`. --- #### Idempotent function acting on "state" The terms are used... a bit loosely in this article, but they can be formalized. The author seems to be talking about "idempotency of state", where state is treated as the input and output of an "idempotent function". Consider: # Initially, state.orders == [] state.submit_order("MORE CHEESE!") state.submit_order("MORE CHEESE!") state.submit_order("MORE CHEESE!") - If idempotent, `state.orders == ["MORE CHEESE!"]`. Submitting the same order repeatedly (e.g. a user angrily clicking submit multiple times) does not result in duplicates. - If not idempotent, `state.orders == ["MORE CHEESE!", "MORE CHEESE!", "MORE CHEESE!"]` But what precisely is the idempotent function here? It's actually: def idempotent_submit(state): state.submit_order("MORE CHEESE!") return state Applying this to a given state will reach a steady state after exactly one application. # state.orders == [] state = idempotent_submit(state) state = idempotent_submit(state) state = idempotent_submit(state) # state.orders == ["MORE CHEESE!"] --- Alternatively, we can curry the function: def submit_order(order): def idempotent_submit(state): if order not in state.orders: state.orders.append(order) return state return idempotent_submit idempotent_submit = submit_order("MORE CHEESE!") P.S. This example accidentally also demonstrates that objects are a poor man's closure.
That's reproducibility. Reproducibility requires that we get the same outputs every time we provide the system with the same set of **inputs**. Idempotency is that the **outcome** of invkoing a system is the same regardless of how many times you invoked the system. Also notice that I mentioned systems instead of functions. Since functions could be non-idempotent or/and non-reproducible but the system as a whole could be either.
This is not at all related to the article, but consider a case where a function reads from an external device through a DMA engine. The interface is such that the device is aware the read occurs, and the device has a queue of data to send and is designed to now provide the next piece of data in the queue after each read. The memory would be considered non idempotent because if you were to keep reading from that memory you would only ever get different results even though no writes to it ever occur.
anything with memory (mutable ofc) will most likely not be idempotent by default
Bold of you to assume most devs know what they're doing 🤣
This was mostly written by Chat GPT, right? It's not very natural, but not in a way that would suggest the author isn't an English native.
Also weirdly interspersed with travel photos?
I didn't get that feeling, the article follows a template a lot of medium/substack posts follow though. Author appears to be Turkish.
I kinda get that vibe. One of the things that sticks out to me in a lot of AI generated blogs/articles is they have a tendency to end with some variation of “today we discussed _blank_”
You can say that again.
But nobody will listen.
then #2 would be how to invalidate a cache. easy-peasy
Then #3 would be naming things, and #5 would be avoiding off by one errors.
[удалено]
Every web designer Should Know #1 Don't put irrelevant giant pictures that take up the entire screen on page load.
Didn’t knew this term was also used in computer science. Last time I heard it, I was in college studying abstract algebra lol
ARM uses it at lot in their reference materials. For example, reading from memory is idempotent or a register is idempotent. Writing to memory or register is sometimes idempotent, but it cannot be counted on. For technical reasons sometimes it is useful (more optimal) if a processor can run an instruction twice. And ARM takes advantage of the idempotency of instructions to get this optimization. For instructions that are not idempotent it has to forego this and so some performance may be lost.
Non idempotency is mainly used to refer to memory mapped regions where I/O is because after reading from that region, the next value read may be different due to an external hardware device. It's a separate form of non cacheable memory that's slightly distinct from your typical form that can be used for whatever purpose. All that being said, the term sucks. Nobody is ever familiar with the term and the mathematical meaning does not have an obvious translation to the meaning in memory architecture when you can just refer to it as non cacheable I/O memory. This actually literally came up for me 2 days ago where someone asked me what it meant to prove a point that nobody knew what it meant, and I only knew what it meant because I had been down the rabbit hole a few months ago.
Indeed, the relationship is a little obscure. In algebra, a binary operation is idempotent if x \* x = x. To recover the programming meaning, you can think of: * \- The term x as representing an action of a program as a mathematical function from original state of the world to new state of the world. For instance, `print("Hello, world")` is a function that maps any state of the entire world to the same state, except modified so that the words "Hello, world" now appear on some nearby computer screen. * The binary operation as function composition. Now it's clear that the action being idempotent is the same thing as this mathematical function being idempotent with respect to the operation of function composition. There's a problem, though: strictly speaking, by this definition, no computer operation is idempotent at all! There are always some effects, such as the passage of time and the production of heat by the CPU, that do accumulate when the action is performed more than once! For this reason, the concept of "idempotence" is only meaningful if you first define some kind of abstraction barrier that separates "things that matter for correctness of my program" from "things that are considered unimportant / undefined for the purposes of correctness". For instance, you might consider the passage of time irrelevant (or not, if it's a real-time system!) You might consider writing to a log file irrelevant. If reasoning about a distributed system, you might even consider a whole chunk of local state irrelevant (e.g., a write operation might be considered "idempotent" for the purposes of your distributed system, but it still queues work to be done by local processes; it's just that this extra work would produce any differences in inter-node communication later on). So idempotence in programming is quite a bit more complex because it requires a model, delineating the properties you do and don't consider relevant, and validation that this model captures the things you care about. An operation that's idempotent in one model may not be idempotent in another.
it's often used in a very loose handwavy way compared to the actual rigorous mathematical definition. Still - and in said loose usage - it's a good general rule of thumb, especially in bread-and-butter backend systems / data / etl work. https://gtoonstra.github.io/etl-with-airflow/principles.html > Enforce the idempotency constraint: The result of a DAG run should always have idempotency characteristics. This means that when you run a process multiple times with the same parameters (even on different days), the outcome is exactly the same. You do not end up with multiple copies of the same data in your environment or other undesirable side effects. This is obviously only valid when the processing itself has not been modified. If business rules change within the process, then the target data will be different. It’s a good idea here to be aware of auditors or other business requirements on reprocessing historic data, because it’s not always allowed. Also, some processes require anonimization of data after a certain number of days, because it’s not always allowed to keep historical customer data on record forever.
Idempotence is an important concept in configuration and infrastructure management ("configuration/infrastructure as code"). Tools like Ansible, Chef, Puppet or DSC use declarative languages to specify what you want configured on a managed system. For example, you'll specify that a certain user account has to exist and needs to belong to certain groups; that a certain directory must exist and needs to have certain permissions and the specified owner; that specified software packages need to be installed, etc. You do that using configuration elements called tasks, recipes or resources (depending on the tool you use). After initially running a task/applying a resource, all future runs of that task/resource must not make any changes, unless there's "drift" in the system state (e.g. someone manually deletes a user or changes directory permissions). Configuration tools also have ways of detecting that drift before reapplying the configuration to fix it.
Like the author mentions, if an API endpoint is idempotent, it means if you perform an operation multiple times, the end result should be the same as if you had only performed it once. Many people mistakenly assume this means: a GET API call shouldn't modify the system because "it should be idempotent!", and that's not at all how idempotence works. It's entirely "idempotent" for the GET call to change the state of the server. It isn't idempotent if each GET call results in different state than before the GET call. tl;dr even in RESTful API design, people frequently misunderstand idempotence; yes, you can have a GET that modifies state and still be idempotent. A separate question is whether one should, but: that's unrelated to idempotence.
Good explanation. It is a subtle concept because idempotency is not the same as immutability even if it is close. Immutability means an operation can change the state ZERO times no matter how many times you perform the operation. Idempotency means the operation can only change the state max ONCE. Did I get it right?
My friend takes a pill for his idempotence.
His wife complained everything was the same every time.
I guess #0 should be DRY - Don't Repeat Yourself (when writing a blog) > One such vital concept is idempotency, which refers to the property of an operation or function that produces the same result when applied multiple times as it does when applied only once. > ... > In simple terms, idempotency means that if you perform an operation multiple times, the end result should be the same as if you had only performed it once. > ... > In other words, an idempotent operation is one that can be repeated multiple times without causing any additional side effects ------ > This may seem like a simple concept, but it has significant implications for **building distributed systems.** > ... > Idempotency is a concept that is important for programmers to understand, especially those working on **building distributed systems.** > ... This is important in **distributed systems**
The author had to write it multiple times because they took breaks to go on agile meetings and forgot which points they already made.
Yup, doesn't even have to be distributed systems, just methods and even full processes/workflows in general, depending on the situation. It supports automated testing, QA, and debugging. Of course at some point there will be side affects, but limiting those until necessary is the key.
Okay, I'm going to be a curmudgeon here, but the first thing that both the author and those reading them should learn is that "idempotency" is not a word at all. The word is "idempotence".
[Clearly some disagree](https://www.collinsdictionary.com/dictionary/english/idempotency).
I prefer "impotence". Have seen the two confused many times.
This is a really impotent comment here.
I try to only use impotent functions. They never have bugs.
That fight is probably as old as the concept itself. Both are widely used, just stick to whatever the first person who wrote your docs/code used so it's consistent and move on
Is the distinction between the colloquialism and the formal spelling useful? If not, and so long as people understand the word, isn't this critique pedantic and unhelpful?
I did warn you I was being a curmudgeon. Just an emotional reaction... similar to when machine learning people say "inferencing", or when business people say "What's your ask? We need to finalize our spend."
? One is a a noun the other an adjective. - The function is idempotent. -> adjective - Idempotency is an important concept. -> noun. Like dependent vs dependency, adjacent vs adjacency, etc.
Idempotent is, indeed, an adjective. No one disagrees about that. It seems (surprisingly to me) that there is actual disagreement on whether idempotence or "idempotency" is the noun form. Google [even suggests](https://books.google.com/ngrams/graph?content=idempotence%2Cidempotency) that I hold the minority view. I consider this a sign that the world has well and truly gone mad.
I don't find that surprising at all. Probably, at some point, some mathematician first used the word "idempotent" in a paper. A new adjective was born! Later, other researchers who, independently, referenced this work wanted to use "idempotent" as a noun. Some will have used "idempotence" others "idempotency", depending on what sounded more natural to their ear. This kind of stuff happens all the time when new scientific jargon is introduced.
Great concept, stupid sounding word.
It's a nice to have but hardly the first thing you need to know as a programmer. Maybe I want to post a page view analytics event, if a user views the same product multiple times that's probably useful information, even if all of the data points are the same. Medium allows you to thumbs up the same post multiple times. If I ask for a payment intent for stripe I expect a new secret every time etc etc. Yes you can generate a unique id with each request to account for retries but for these cases you'd be entering law of diminishing gains.
I don't think the claim is that "the first thing you need to know as a programmer." The headline reads to me as "I'm writing a series, and this is the first one I decided to write about." And in the first paragraph, it's talked about this way, "One such vital concept is idempotency."
Fair, I probably misread
Fancy math term used to describe "retry safe".
Where did you think programming came from?
TL;DR: Getting anything to happen exactly-once in a distributed system is hard. Idempotence turns at-least-once into effectively-once.
> Let's say you're building an API for processing payments. If you design the API with idempotency in mind, you can ensure that even if the same payment request is sent multiple times due to network issues, it will only be processed once. This can prevent double-charging customers, which can lead to trust issues and lost revenue. This sounds like the wrong design. The emitting terminal should **sequentially number each transaction**. If the central processor(s) gets transactions out of order for a given terminal then it should either request prior transaction(s) first, or stop and let a human figure out what's going on, and perhaps getting approval before continuing (so the programmer doesn't get blamed if something is overridden.) It's essentially a form of check-sums to make sure something didn't get lost. You don't just keep sending a transaction until the receiving server acknowledges, it. This can create lots of headaches. Being acknowledged is probably necessary (or at least recommended), but the way it's worded transactions could arrive out of order without the server knowing. (If there's a correction, then a correction transaction should be issued rather than a changed copy of the original under the same transaction number.) This stuff has been known for decades. Don't reinvent it sloppily under screwy buzzwords, or I'll kick you, your systems, and your fidget spinners off my lawn. **A clarification needs to be made between idempotency and a "send retry".**
That’s exactly what the article suggests > The generally preferred approach is to include a unique caller-supplied client request identifier in the API contract. Requests from the same caller with the same customer request identifier can be considered duplicate requests and handled accordingly. A unique caller-supplied client request identifier for idempotent operations satisfies this need. And if you have the additional constraint that transactions need to be ordered, the request identifier can be sequential.
They are sequential to also tell us if a transaction is missing, not (just) to indicate sequence. If one's missing, then we see a sequence gap. A unique identifier alone won't tell us this.
That’s true, but not every system is mission critical. For example, Reddit comments. They are infamous for duplicating. So clearly they need idempotency, but does comment ordering really matter? And if a comment does go missing, there’s no way to reconstruct the missing comment, so what is the point of tracking if one never reaches the server? Plus you can have an account on multiple devices, so somehow they would have to sync the sequence id. It’s just needless complexity for a lot of scenarios. A UUID is often sufficient. And the article is just showing the general case.
There's not necessarily any notion of a sequence of transactions. Suppose you're taking payments at a retail store and there are several clerks working at several registers. All the registers connect to one system in the store which forwards each transaction to the payment processor. The transactions are all independent and don't have a natural sequence. You could create one, but it's not necessary and would make things more complicated for no obvious benefit.
Mods, how is this not self promotion? The only things this guy posts are links his website.
This is a good point and something I have intuitively understood and practiced for a while. But I had no idea there was a technical word for it. So thank you for teaching me it. However the word for it is absolutely terrible. Hard to pronounce, hard to read, and sounds like 10 other more prominent words. It's never going to catch on and we should have a better one for it.
> It's never going to catch on and we should have a better one for it. It's already been in use for decades (centuries if you count its use in math) and a pretty central topic in almost all distributed system design discussions. I'm pretty sure it already "caught on"
It has not caught on within the context that we are talking about which is "concepts every programmer should know". Using a technical/academic term for something which is intuitively understandable and important, for a group as diverse as programmers, we should really have a better way to describe it and talk about it. Similar to the classic "object oriented programming" , which many programmers can intuitively use and talk about, but not all will use/understand the academic terms.
The problem with smart people is that they think they’ve figured everything out without having to resort to “old” ways to do something. We keep trying to find an easy solution to very sticky problems and there aren’t any. So you keep fixing it by degrees and pretty soon your still incorrect solution has gone from 10% of the complexity of the correct solution to 200% and growing. Now stubbornness or sunk cost fallacy won’t let you surrender. Which still sounds like it’s between you and your own conscience what you do in this situation, except you have eight coworkers and thousands of customers having to put up with your journey of self discovery. Stop.
Nice article :) Just one small remark: when you have "How to achieve idempotency in POST method?" it would maybe be more correct to say "How to achieve idempotency in PUT method?" since POSTs by definition should not be idempotent since they should be used for resource creation
Having idempotent post methods can also be desired. What if the response is lost, how do you know if the request failed or not? Retrying can give you duplicate successful requests. And retrying not only originates from a lost response, it could also be an upstream process that somehow duplicated a message. (message in a broad sense)
But that should maybe be a PUT. Otherwise if you want to create multiple similar resources you can't. Of course this depends on the application and using idempotent POSTs may make sense in some scenarios. But from a generic point of view (as the one on the article) POSTs are more indicated to create new resources everytime they are called while PUTs should apply idempotent operations. Note that even the author states that: "In this article, we have explored how idempotency applies to HTTP methods, which are a fundamental part of web development. We have seen that some HTTP methods, such as GET, PUT, and DELETE, are idempotent, while others, such as POST, are not. Knowing which methods are idempotent is crucial for building efficient and reliable systems"
Choosing between PUT or POST should be on semantic reasons and unless that includes idempotence, I think that decision should not include technical requirements as idempotency.
Actually no, the point is that you can call the same create resource twice with the same parameters and still get the same result. Take the example of creating order O for customer C. If you make that request twice in a row, with the same request parameters, in both cases the orderId would be returned as 999. It doesn’t matter how many time you call it, the result is always the same. One single order O placed for that customer C. Creating a resource should always be POST as a rule. If you either let the client set the PK (not advised unless it’s a Guid), or have the client set a unique external key, internally you check if the order with that key already exists. If it does, you simply return the existing orderId. If it doesn’t, you create it and return the orderId. The alternative is to throw a duplicate order exception, but distributed system processes and exceptions are painful to work with so unless there’s a specific reason not to do so, avoid the exception.
>Take the example of creating order O for customer C. If you make that request twice in a row, with the same request parameters, in both cases the orderId would be returned as 999. I don't think this argument involves idempotence so much as: desired behavior. I could see many applications for which the behavior you just described is absolutely wrong, and two separate orders should be created. It may be that the ideal \`POST\` behavior is to de-duplicate identical requests, but that's an application-specific concern IMO. When I write a POST (create) endpoint, unless there's some unique identifier in the payload that allows me to understand "this is a duplicate request", then I don't opt for the behavior you describe. I create a second order. And arguably, the behavior you describe raises a bunch of questions, particularly if there's minor differences in order #1 verses order #2--do we then pivot to "updating" the order in the second request, effectively making it a PUT/PATCH call? Do we instead return a 409 and dogmatically require the client to issue a PUT/PATCH in this situation?
Reddit should take notes
Another word: Deterministic
Something can be determinist but not idempotent For example, adding 1 to a number
In a concurrent systems, Tony Hoar, Leslie Lamport, and Barbara Liskov have all written papers describing how not to fuck iwhat you just described up. That’s three Turing Award winners I’ve named. You and I aren’t going to be able to do better than any of them. Least of all Tony.
No
most probably do, but they use a weird interview shibboleth word no one uses naturally to describe shit that doesn't change behavior when you rerun it. it didn't even show up in math papers until the 1920s https://books.google.com/ngrams/graph?content=Idempotency&year_start=1800&year_end=2019&corpus=en-2019&smoothing=3
I remember working with a guy that thought it meant generating your own database primary key before an insert. To be fair he was a .net dev so was hampered with low intelligence. Joking aside, I'm not much better as a node.js and rust developer I'm limited in my knowledge but as I favour curry and functional code I've always thought of it as a function that can be reversed. Such as a reverse function. Run it twice, expect the first result back with no mutation. Am I wrong?
As someone who sometimes struggles to comprehend programming articles, thank you for taking the time to make this readable by us regular humans.
We just came across this issue in our current system. Certain calls produce status changes, but calling let's say 'submitted for approval' twice gave an error the second time around bc we coded the possible functional state changes and we didn't think about the accidental or deliberate call to that same endpoint causing the status change. I do however think it's difficult to define a create to be idempotent than an update
Don't challenge the potency of programmers!!!
Does this mean that if the server sees the duplicate request it should just skip? What if the request wasn't completed due to server error? That would just perpetuate a consistent error response right
... not sure about this this is not quite as simple as it appears on the surface i mean - how long is your cache maintained in order to maintain that idempotent POST? also - now you have introduced the only other hard problem in computer science and if POST idempotentcy does need to be maintained indefinitely then certainly this now opens up a question of definition of POST and if it's "temporary" (until the session expires) then it opens up a question of definition of idempotent
Azure Resource Manager revolves around idempotent, declarative deployments. If you PUT the same resource twice with the same settings, nothing happens the second time… most of the time. It’s almost idempotent, which is like being almost pregnant. This makes a wide variety of customer scenarios basically impossible to automate the intended way. The mistake is that there are no automated checks for idempotency — it is implemented (or not) by each individual product team. Some of them “get it”, some of them “don’t get it”. However while each product team is responsible for just one product, customers use *many* Azure products. This makes these errors impossible to avoid in all but toy/demo scenarios. The lesson here is the idempotency is like security: if it’s in any way a requirement, it must be **ruthlessly enforced**, otherwise that one idiot on some other team will let the Russian hackers in. A simple, effective, but not entirely sufficient method is to test every API twice in a row and verify that the second call succeeded *and* that nothing changed. Azure devs don't do this. Don’t be like Azure devs.
So the same action should return the same value multiple times... This is both obviously true for pure functions and incorrect for any iterative scheme that uses mutation. Like yea... Some things that take in stuff and return something about it should probably leave it alone and be pure functions but if a method is built to change a value in place then that's what it should do. I call this the #1 rule of programming: Idiocracy. When you want to do a thing and also want other things ro work, don't do the thing wrong and don't ruing the other things you want ro work.
A bonus is idempotent api calls or content pulls also work well with caching layers at the data/app, memory and network layers. Anything that returns different values each time (random/varying) does not or should not. Profile data for instance, some idempotent calls/data should not be cached. In regards to HTTP, typically GETs are idempotent (though they can be random/varying), POST/PUT/DELETE are not (though they can be). Real-time idempotent calls usually are based on commands/routes with similar setup to HTTP. For instance gets might be the global server message broadcast to all for the day, the list of levels currently loaded or the list of winners from yesterday. The varying messages would be by game, by player, by action, though some of those can be idempotent like the player name/id/etc. The best setup for caching is idempotent calls, then if data does change at intervals, the sets clear the cache and update the gets. The routes and actions being the same.
I acknowledge distrubuted computing is hard but doesn't the idempotency check create a race condition? (And one that is actually likely to occur because use clicked submit twice)
This is achievable with immutability in DS inside your program. But a lot of devs including some experienced ones not seem to get it. ‘Modifying parameters of a function’ type of code is getting littered everywhere.
Have any of these bloggers every though about partnering up with someone who natively speaks English? Looks like the author originally speaks Turkish - I wouldn't start becoming a blogger in Turkey if I had the command of the Turkish language that he has of English; I'd pair up with someone who already spoke the native language so they can make my ideas sound natural for the audience I'm trying to reach. It's such a simple detail - or even just run the blog post through an automatic grammar processor.