I used to travel doing optical networking deployments between data centers. Got to see a good 35+ states and even some foreign countries over 4 years or so but also completely burnt out. Telecom is a pretty miserable industry.
Most of the work in/around DCs is contract of some sort. Local structured cabling firms, electricians doing stints of various lengths, security, etc. Some of it good and lucrative, some of it not.
I did the same in europe and burnt out with my colleague after ~3 years.
Working and driving at night, barely enough sleep and overtime compensation during the time the body needs to recover was the worst.
I probably aged 10 years in that time.
Data center construction is consuming all of the tradespeople who usually do work in homes and causing price spikes for contracting similar to what we are seeing for hardware prices. I’m guessing those workers are doing at least OK for themselves ;)
I started out in a "data centre" in 1998/1999 in the North West of England. Things are radically different now.
When I joined that ISP we had ~2000 subscribers, and one modem rack that could take 30 concurrent 33.6k dial-ins. The company had just been acquired by a small independent telco in Cheshire, headquartered in a small town that is most famous for a weekly farmers market and an annual steam rally.
We shared that headquarters at first with the founder's other business - a dating agency for rich people in later years - and slowly took over.
The building was a farmhouse of sorts. We'd turned into a rabbit warren of offices, and then, we decided to expand.
First we turned one of the offices into a "machine room". We didn't even bother with fire suppression that first time, we just ripped up the carpet and laid some laminate, put some 19" racks in and got the power upgraded. The air-con was insufficient in that room and the following summer I had to wheel in portable units and had a Cisco router in there report that the inlet fan had a temperature of ~80C. That was around the same time the chief of staff was asking us why we wore jeans and t-shirts and couldn't we wear suits? No, no we could not wear suits.
Bit by bit we took over more rooms through the building, we started working in a portacabin in the car park (where the neighbouring farmer's sheepdog would join us for lunch).
At some point we needed to install ~20k modems, (Nortel CVX1800s - 56kbps had arrived!), and the telephone switch to run it, so somebody bought a second hand DMS100 that went into another room we had converted.
By that point the dating agency had moved out.
We needed another ~100 servers as well to manage email, web hosting, and so on. Those came in parts (motherboards, cases, RAM sticks, hard drives - about £500k worth in 1999 money), and we had to build them and install BSD on them (first OpenBSD, then FreeBSD, long story). We also had to make our own ethernet cables, which is where I learned I can't wire ethernet cables if my life depended on it.
Within 12 months from when we started, we were handling 750,000 dial-in customers.
Around September of 1999 I left and that was when the company had heard about a custom built data centre that was up for sale in Milton Keynes, so decided to move everything there. Around that time they were pouring concrete under the floors of all those rooms we'd thrown racks in because the timber floor joists (!) were starting to give way under the weight of the kit.
That company got acquired and then went through a weird route, but basically it's still going as a major UK ISP. I'll keep the name private to protect the guilty. :)
2000 subscribers, and one modem rack to that and a business still operating 27 years later. A lot of sausage rolls, plenty of sticking plasters for the cuts from those shitty cases, and almost every day ending with a couple of pints in a pub somewhere.
One of my old colleagues and I often think back to that being one of the best years of work in our lives. But... data centre work is horrid, dirty, hot, and underpaid for what it is. But you can - sometimes - be part of something kind of cool and fun, and something very few people get to see or understand.
But there's a reason I've stuck to software ever since. :)
This made me smile, reminds me of my 90s, early 00s in web development & hosting. Where tiny offices above shops spread out as the business grew; racks were thrown in cupboards; the noise was constant and the heat was unbearable in the summer; the pigeons nesting on the shit covered Victorian window sill had the best seat in the office and nobody really knew what they were doing but we were all incredibly cocky about being wunderkids working on the coolest new stuff that nobody else understood. Good times!
>We also had to make our own ethernet cables, which is where I learned I can't wire ethernet cables if my life depended on it.
Are you me? would always take me about 10 attempts to even get one where all the pins would light up on a tester. But not good enough for high-speed data.
Kids these days, don't know when they were born. They have push-through RJ45 now, piece of piss and you never have one shorter than the other.
I don't miss on-site/DC infrastructure anymore. No more having to squeeze into tiny spaces, balance a lot of expensive kit on your head, and blood donations to the server gods.
Compute is so ubiquitous now I only have like 2 nodes in my lab anyway
You'll often find a lot of engineers, and in my experience, a lot of former navy nuke personnel, overseeing the power and facilities at large data centers. There also tend to be a number of others with engineering backgrounds working in data center operations even if a lot of their work may not be what would commonly fall under "engineering". Technicians who perform some of the break/fix and routine maintenance do tend to be lower skilled but, again, it's not uncommon to find advanced degrees. I knew someone with a Phd in Chemistry working as an entry level data center technician because they were simply looking for something different outside of academia and it was a career that had rapid advancement potential.
Same is true for software "engineers". We don't take a professional exam or adhere to any professional licensing. We aren't legally engineers in some states/nations.
Don't know about US, but I work within European Datacenter where we intend to have excess heat to be used for residential building heating [1]. You must be in proximity of the consumer for this to be possible. Not necessarily center is needed, but even then, DCs can be pretty :)
For example our office is heated using heatpump that has >22C of water heated by equipment as a source.
> excess heat to be used for residential building heating
In the late 90s I worked at an insurance company in leafy Surrey[1]. I was told that they used to use their IBM mainframe to heat the swimming pool[2] effectively for free; when they upgraded they had to buy and run a heating system for the pool as the new mainframe didn't pump out enough heat.
[1] In England, south west suburbs of London to be precise.
[2] yes it had an indoor pool, badminton courts, outdoor tennis courts and other facilities - a great place to work.
It is crazy how you see so many datacenters getting built in places of high land value such as say northern virginia. Why not just build them in west virginia and build twice as many for the same spend? They'd probably even let you burn coal to power them.
I'm guessing it's to do with regulation. There's probably some policy document that says a certain org needs to be able to access the physical servers within a certain amount of time, and so they need to be nearby, even though nobody ever does the accessing.
DC would be the kind of place with a lot of government contractors that would have this kind of stipulation, is my guess.
West Virginia also has a lot of coal. I wonder why the current Admin doesn't encourage DCs to be built next to coal. Coal communities don't care much for any regulations, DC development should be perfectly fine. Hyperscalers can brag about 100% coal powered AI.
I suspect it's driven by availability of existing grid infrastructure (so they don't have to wait for upgrades) and ease of permitting; the contents of the building and indeed running costs are so high that the land value is much less relevant.
Nearby skilled labor, police/fire infrastructure willing to handle it, and ease of demo/tours/sales for clients to install their racks. Also much more stable power.
There is a lot of electrical supply and demand is high plus incentives from Virginia itself and proximity to DC. It’s already a data center dense market
At my first job in a bank there was an old-timer whose job had been to fly to every trading desk the bank had in other countries to install software updates. He would do a round the world trip 1-2 times a year with multiple duplicate boxes of floppies during 3-4 weeks.
TL;DR: you walk around smelling and listening for issues. Most of the staff on site are doing the same thing; the rest are private security hired by customers housing within the DC. You make between $66K and $120K/annual.
There, saved you from reading a story that isn't really informative and only was written because of the politics surrounding DCs are keeping them in the news.
TBH it was informative for me because I've got family who applied for data center jobs. But I'll admit that you captured it all in your blurb. The article has a lot of fluff.
Speaking of DCs, is there evidence that they are sucking up water? What about deteriorating air quality around the buildings? Could they be causing major environmental issues? It seems they mostly have closed loop systems a la CPU coolers and they consume a lot of electricity that's already been earmarked for them. What about air quality and air humidity?
I worked at Google datacenters for over a decade. It was a huge ordeal to request running generators because our entire ECS (environment control services) team was geared towards NOT having to run them. I know AWS and Microsoft both have a similar situation. The referenced article just seems like senseless fear mongering.
There have been suggestions that the DC operators themselves have been astroturfing articles about water use to steer discussions away from the other issues.
I used to travel doing optical networking deployments between data centers. Got to see a good 35+ states and even some foreign countries over 4 years or so but also completely burnt out. Telecom is a pretty miserable industry.
Most of the work in/around DCs is contract of some sort. Local structured cabling firms, electricians doing stints of various lengths, security, etc. Some of it good and lucrative, some of it not.
I did the same in europe and burnt out with my colleague after ~3 years. Working and driving at night, barely enough sleep and overtime compensation during the time the body needs to recover was the worst. I probably aged 10 years in that time.
A bit like this? https://www.youtube.com/watch?v=K8KXXJH8Zy4
Data center construction is consuming all of the tradespeople who usually do work in homes and causing price spikes for contracting similar to what we are seeing for hardware prices. I’m guessing those workers are doing at least OK for themselves ;)
I started out in a "data centre" in 1998/1999 in the North West of England. Things are radically different now.
When I joined that ISP we had ~2000 subscribers, and one modem rack that could take 30 concurrent 33.6k dial-ins. The company had just been acquired by a small independent telco in Cheshire, headquartered in a small town that is most famous for a weekly farmers market and an annual steam rally.
We shared that headquarters at first with the founder's other business - a dating agency for rich people in later years - and slowly took over.
The building was a farmhouse of sorts. We'd turned into a rabbit warren of offices, and then, we decided to expand.
First we turned one of the offices into a "machine room". We didn't even bother with fire suppression that first time, we just ripped up the carpet and laid some laminate, put some 19" racks in and got the power upgraded. The air-con was insufficient in that room and the following summer I had to wheel in portable units and had a Cisco router in there report that the inlet fan had a temperature of ~80C. That was around the same time the chief of staff was asking us why we wore jeans and t-shirts and couldn't we wear suits? No, no we could not wear suits.
Bit by bit we took over more rooms through the building, we started working in a portacabin in the car park (where the neighbouring farmer's sheepdog would join us for lunch).
At some point we needed to install ~20k modems, (Nortel CVX1800s - 56kbps had arrived!), and the telephone switch to run it, so somebody bought a second hand DMS100 that went into another room we had converted.
By that point the dating agency had moved out.
We needed another ~100 servers as well to manage email, web hosting, and so on. Those came in parts (motherboards, cases, RAM sticks, hard drives - about £500k worth in 1999 money), and we had to build them and install BSD on them (first OpenBSD, then FreeBSD, long story). We also had to make our own ethernet cables, which is where I learned I can't wire ethernet cables if my life depended on it.
Within 12 months from when we started, we were handling 750,000 dial-in customers.
Around September of 1999 I left and that was when the company had heard about a custom built data centre that was up for sale in Milton Keynes, so decided to move everything there. Around that time they were pouring concrete under the floors of all those rooms we'd thrown racks in because the timber floor joists (!) were starting to give way under the weight of the kit.
That company got acquired and then went through a weird route, but basically it's still going as a major UK ISP. I'll keep the name private to protect the guilty. :)
2000 subscribers, and one modem rack to that and a business still operating 27 years later. A lot of sausage rolls, plenty of sticking plasters for the cuts from those shitty cases, and almost every day ending with a couple of pints in a pub somewhere.
One of my old colleagues and I often think back to that being one of the best years of work in our lives. But... data centre work is horrid, dirty, hot, and underpaid for what it is. But you can - sometimes - be part of something kind of cool and fun, and something very few people get to see or understand.
But there's a reason I've stuck to software ever since. :)
This made me smile, reminds me of my 90s, early 00s in web development & hosting. Where tiny offices above shops spread out as the business grew; racks were thrown in cupboards; the noise was constant and the heat was unbearable in the summer; the pigeons nesting on the shit covered Victorian window sill had the best seat in the office and nobody really knew what they were doing but we were all incredibly cocky about being wunderkids working on the coolest new stuff that nobody else understood. Good times!
>We also had to make our own ethernet cables, which is where I learned I can't wire ethernet cables if my life depended on it.
Are you me? would always take me about 10 attempts to even get one where all the pins would light up on a tester. But not good enough for high-speed data.
Kids these days, don't know when they were born. They have push-through RJ45 now, piece of piss and you never have one shorter than the other.
I don't miss on-site/DC infrastructure anymore. No more having to squeeze into tiny spaces, balance a lot of expensive kit on your head, and blood donations to the server gods.
Compute is so ubiquitous now I only have like 2 nodes in my lab anyway
> blood donations to the server gods
It really bothered me that for how much enterprise gear costs, the manufacturers couldn't be bothered to deburr the edges of rails/servers/etc.
Engineers, really? Maybe technicians ?
You'll often find a lot of engineers, and in my experience, a lot of former navy nuke personnel, overseeing the power and facilities at large data centers. There also tend to be a number of others with engineering backgrounds working in data center operations even if a lot of their work may not be what would commonly fall under "engineering". Technicians who perform some of the break/fix and routine maintenance do tend to be lower skilled but, again, it's not uncommon to find advanced degrees. I knew someone with a Phd in Chemistry working as an entry level data center technician because they were simply looking for something different outside of academia and it was a career that had rapid advancement potential.
Same is true for software "engineers". We don't take a professional exam or adhere to any professional licensing. We aren't legally engineers in some states/nations.
next you're going to tell me that the guys who drive locomotives aren't engineers
Unlike the USA where anyone can be an "engineer", there are countries where words mean something and terms are protected: https://engineerscanada.ca/become-an-engineer/use-of-profess...
The smell is nostalgic.
It's bizarre that Digital Realty wants to put a DC in a town centre. Surely the latency disadvantage from being outside city limits is neligible.
Don't know about US, but I work within European Datacenter where we intend to have excess heat to be used for residential building heating [1]. You must be in proximity of the consumer for this to be possible. Not necessarily center is needed, but even then, DCs can be pretty :)
For example our office is heated using heatpump that has >22C of water heated by equipment as a source.
https://delska.com/about/news-resources/delska-newsroom/pres...
> excess heat to be used for residential building heating
In the late 90s I worked at an insurance company in leafy Surrey[1]. I was told that they used to use their IBM mainframe to heat the swimming pool[2] effectively for free; when they upgraded they had to buy and run a heating system for the pool as the new mainframe didn't pump out enough heat.
[1] In England, south west suburbs of London to be precise.
[2] yes it had an indoor pool, badminton courts, outdoor tennis courts and other facilities - a great place to work.
It is crazy how you see so many datacenters getting built in places of high land value such as say northern virginia. Why not just build them in west virginia and build twice as many for the same spend? They'd probably even let you burn coal to power them.
I'm guessing it's to do with regulation. There's probably some policy document that says a certain org needs to be able to access the physical servers within a certain amount of time, and so they need to be nearby, even though nobody ever does the accessing.
DC would be the kind of place with a lot of government contractors that would have this kind of stipulation, is my guess.
West Virginia also has a lot of coal. I wonder why the current Admin doesn't encourage DCs to be built next to coal. Coal communities don't care much for any regulations, DC development should be perfectly fine. Hyperscalers can brag about 100% coal powered AI.
When it all goes bust, at least you'll have some real estate.
I suspect it's driven by availability of existing grid infrastructure (so they don't have to wait for upgrades) and ease of permitting; the contents of the building and indeed running costs are so high that the land value is much less relevant.
Nearby skilled labor, police/fire infrastructure willing to handle it, and ease of demo/tours/sales for clients to install their racks. Also much more stable power.
There is a lot of electrical supply and demand is high plus incentives from Virginia itself and proximity to DC. It’s already a data center dense market
https://archive.vn/Myhqh
At my first job in a bank there was an old-timer whose job had been to fly to every trading desk the bank had in other countries to install software updates. He would do a round the world trip 1-2 times a year with multiple duplicate boxes of floppies during 3-4 weeks.
TL;DR: you walk around smelling and listening for issues. Most of the staff on site are doing the same thing; the rest are private security hired by customers housing within the DC. You make between $66K and $120K/annual.
There, saved you from reading a story that isn't really informative and only was written because of the politics surrounding DCs are keeping them in the news.
TBH it was informative for me because I've got family who applied for data center jobs. But I'll admit that you captured it all in your blurb. The article has a lot of fluff.
Speaking of DCs, is there evidence that they are sucking up water? What about deteriorating air quality around the buildings? Could they be causing major environmental issues? It seems they mostly have closed loop systems a la CPU coolers and they consume a lot of electricity that's already been earmarked for them. What about air quality and air humidity?
Yes, there's a lot of evidence on all of those things:
https://news.vcu.edu/article/northern-virginia-data-center-a...
I worked at Google datacenters for over a decade. It was a huge ordeal to request running generators because our entire ECS (environment control services) team was geared towards NOT having to run them. I know AWS and Microsoft both have a similar situation. The referenced article just seems like senseless fear mongering.
There have been suggestions that the DC operators themselves have been astroturfing articles about water use to steer discussions away from the other issues.
You don't have to downvote me. I am genuinely interested in the impact. I just want to learn.