What is your biggest or funniest oops moment as a developer?


(Frédéric Harper) #1

Our director of security, @adam_baldwin, published an article this week about a change we made in the publishing mechanism to prevent token to be published my mistake with your npm package.

Brilliant change as I’m the first one who is often doing mistakes like these. So, I’m curious, as a developer, what is the biggest or funniest oops moment like this you’ve done?

Personally, many, many years ago (way before my time at npm), I was doing some C# with database stuff and I forgot to change my connection to a SQL database after fixing something in production. Thinking I was testing something locally, I’ve screwed up the customers’ data. Fortunately for me (and our customer), we had a backup and since it was late in the evening, we didn’t lose any data! Those things are never fun, even if I laugh about it today, but at that time, I was junior; I don’t remember the last time I panicked like that!

So, what is your story?


(Kat Marchán) #2

I published a buggy version of the CLI and someone (who I’m sure was lying) says that it ruined three production servers. :sweat_smile:


(Jeff Lembeck) #3

Back many moons ago, pre-npm as the normal solution for front-end libraries, it was not uncommon to go to the website for that lib, click on the newest version, have it load in your browser window and copy/paste it right into a file.

This era happens to have existed as the same time as a plugin called “Cloud to Butt.”

Well, Underscore was very popular and DocumentCloud was the trademark holder and I definitely went to the site, copied it, pasted it into a file (license and all), and went along my way. A bit later, I realized that I had committed a library by “DocumentButt” into my client’s repo.

They never noticed and I am very good at git. Whew


(Frédéric Harper) #4

Yeah, they were lying:wink:


(Frédéric Harper) #5

I don’t remember this plugin, but I definitely remember having no packages manager and doing what you just wrote… Kids these days, they have it easy: they just need to run npm install :smile:


(Pierre Paul Lefebvre) #6

Spent 3 hours last night trying to understand why the handshake wasn’t happening with my Wireguard server. Turns out I inverted two numbers in one of the configs (51820 -> 58120).


(Frédéric Harper) #7

I had to check twice to see the inverted numbers!


(Chris Dickinson) #8

I, uh, accidentally turned registry.yarnpkg.com into a teapot. Sorry Yarn users! I really didn’t mean to <3

Some background: this last summer we were busily switching CDNs to Cloudflare. Cloudflare has rules restricting where requests can be made – one account on Cloudflare cannot configure internal redirects to another Cloudflare account’s domain. We were unaware of this rule at the time — it didn’t occur to us at the time to check where registry.yarnpkg.com was hosted (It turns out: it’s hosted on Cloudflare and internally redirects to registry.npmjs.org. You might see where this is heading.)

So, we flip the switch to migrate on a Friday afternoon – a low-traffic time for us. After flipping the switch, registry.yarnpkg.com became unusable due to the security restrictions. Cloudflare was super helpful in fixing the bug, which involved allowing the Yarn Cloudflare account to talk to the npm Cloudflare account’s domains. However, Cloudflare advised us to restrict the incoming domains that we allow requests from via the Host header.

In our Cloudflare worker, I add an allow-list that checks the incoming Host header against a list of allowed hosts. Easy-peasy. Somewhat cheekily, I added a 418 I'm a Teapot response for any Host not explicitly allowed (anything that’s not registry.npmjs.org or registry.yargpkg.com, for example). I mean, no one’s ever going to see this, right?

However, some Host headers from registry.yarnpkg.com contained port numbers (registry.yarnpkg.com:443), and thus we went from “The yarn registry url doesn’t work” to “The yarn registry now claims to be a teapot.” Cough.


(Maximilian Antoni) #9

I was responsible for the frontend of an FX trading platform that used push technology to update prices in the browser in real time. I introduced a catch-all error handler that reported client error over XHR. Shortly after, a push message triggered an exception on all clients simultaniously. The flood of messages triggered a deadlock in the logger thread due to another bug. We called it an “attack of self denial”. It brought the whole cluster down.


(Ian Remmel) #10

I was working on a real time messaging service and needed to handle web socket failures. I didn’t realize the close event and the error event both fire when the connection fails to connect (seriously, why would close fire if open hasn’t?). In any case, I put the same recovery code in event handlers for close and error, thus leading to two attempts to reconnect per close. In a scenario where failures were guaranteed, each client would make exponentially increasing reconnect attempts.


(Jon Ege Ronnenberg) #11

Around 10 years ago I worked for a large Telco and we had a homebrewed CMS based on XML where each page template was an XSLT file. There is a lot of restrictions in XSLT, illegal characters, has to be well formed etc. Our staging servers were not auto synchronized with our production servers and we did quick fixes in production. The input was a textarea that depending on browser, IE6 I’m looking at you, could encode characters different from intention.

The issue was that if you somehow introduced malformed XSLT, the parser would break and not just on 1 server but on all servers in our cluster.

On more than one occasion, I introduced malformed XSLT and brought down a 1000+ pages - our entire website went down. I remember always shaking feverishly when doing these kind of edits and the hunt for the often 1 offending character that brought the whole system down.

We also had different versions of Saxon and another parser, can’t remember the name right now, on each server, which added to what kind of XSLT you could use.

A year or two later they changed CMS to Liferay.


(Frédéric Harper) #12

Thanks for the giving the reason as I was reading your story with a strong disapprobation head nodding :laughing:

That’s an interesting story for sure!


(Frédéric Harper) #13

The pleasure of a catch-all!

This is awesome! I mean, the name, not the problem you got, obviously :slight_smile:


(Frédéric Harper) #14

I guess it’s mostly to give you the opportunity to finish any processes when there is no more connection or it’s the end no matter what… Anyhow, I guess you’ll remember how it’s working now :)


(Frédéric Harper) #15

I remember those days where everyone was doing their own CMS… Hell on earth!

Funny enough (or not), it’s often small errors that result in the biggest issues!

It remembers me when I was studying software development and at that time, it was Turbo C++ (if I’m correct, you know, that kind of DOS blue screen “IDE”) that was the cool thing. The compilers weren’t as friendly as today, and I searched for the issue for hours to finally see that I was missing a semicolon at the end of one line… Joy!


(Frédéric Harper) #16

All those stories remembers me when I thought it was a good idea to upgrade my mac with the latest beta version of the OS. It was the day before a keynote I was giving at a conference and I really needed my machine to do my demos. Did you try to reinstall (read download first) macOS (was OS X at that time) from a hotel wifi?

image


(Frédéric Harper) #17

(wait, Discourse doesn’t display gif…)

to see the full extend of my emotion that day --> https://media.giphy.com/media/QB78LMb32YqoE/giphy.gif


(Jon Ege Ronnenberg) #18

I have this feeling everytime I upgrade a mac. Normally I disregard people who won’t upgrade the same as people not taking vaccine but more often than not I advice people against upgrading MacOS, at least 6 months till the bugs has been ironed out. Seems like Windows users are learning the same lesson all over again.

Perhaps antivacination people have point.


(Jon Ege Ronnenberg) #19

I was using Dreamweaver UltraDev at the time you describe, must be late 90`s, which
did nothing for you, except color code table elements in a way that was helpful.

@fharper Jesus, no. You’re talking beginning of 90’s! https://youtu.be/RWavTVo7D3M
We’ll, that’s before my time :slight_smile:

For JavaScript, aptana was easily the best editor together with Firefox firebug. Needles to say, everything sucked compare to today. And do not get me started on venkman.

PS. found a link to venkman


(Jon Ege Ronnenberg) #20

Speaking of offending character, I got a recent story from last year. Some friends of mine run a small crowd funding website and I did a live style guide with 20+ UI elements for them a few years back. Their setup is very manual. They will get a text from a group and copy/paste the text into a .ejs file for each campaign, using node Express to run their website. Then they use the UI blocks to create a unique page but with a unified look n’ feel.

One day, last year, they received a long text from Germany. They copied in the text and begang the design work but the page would not load and was throwing weird errors that they had never seen before. After hours of troubleshooting, reducing the page to find the offending part, they found a letter common on a German keyboard, that broke the template engine! 2018 and this still happens!

I will ask them if they remember what letter, if you’re interested. It must be in a commit somewhere.