Rendered at 18:40:21 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
JohnMakin 2 days ago [-]
> The instinct during an availability incident is to add headroom: raise CPU limits, increase the worker pool, add replicas. That can help with genuine capacity problems. Here, it would only give the retry loop more workers to occupy.
This is something during production issues I have a really difficult time sometimes communicating to peers. "Our services are timing out, we're seeing high latency, increase all resources!" is the knee jerk response, but sometimes, and even often, if the underlying cause of the degradation is something like, a database locking up, increasing workers and giving them more firepower might just make the situation even worse. It happens a lot more than you would think.
elevation 2 days ago [-]
Your comment reminded me of rachaelbythebay.com. While her site seems to be down at the moment, she has shared some useful war stories from debugging similar scenarios. I understand systems better after having read her work.
> My disability is not a tragedy of biology. It is a failure of infrastructure.
Everybody is different. Tall, short, strong, week, two legs, one leg, short sighted or blind. So it makes a lot of sense to point (as a mistake) that our infrastructure favors one type over the others.
> the "Checkout" button is labeled "Button_Graphic_v2_Final"
Couldn’t even be bothered to disguise LLM generated content.
Painful to read.
sahan 18 hours ago [-]
Thanks for the feedback. This was a real incident that I handled. I had to remove any specifics which may have made it generic. I wanted to share the though process and the learnings along the way. Yes, I did use an LLM to polish it up. Could you be specific on why it was painful to read? Was too much text? Was it the tone? Having more concrete examples would help?
This is something during production issues I have a really difficult time sometimes communicating to peers. "Our services are timing out, we're seeing high latency, increase all resources!" is the knee jerk response, but sometimes, and even often, if the underlying cause of the degradation is something like, a database locking up, increasing workers and giving them more firepower might just make the situation even worse. It happens a lot more than you would think.
> My disability is not a tragedy of biology. It is a failure of infrastructure.
Everybody is different. Tall, short, strong, week, two legs, one leg, short sighted or blind. So it makes a lot of sense to point (as a mistake) that our infrastructure favors one type over the others.
> the "Checkout" button is labeled "Button_Graphic_v2_Final"
This is a very good point.
Painful to read.
Thanks for reading nonetheless!