I'm not familiar with your product, just the blog post + cell architecture documentation. Question: the documentation states that basins are assigned to a cell on creation and cannot move. Why choose to have explicit DNS records instead of wildcards with a subdomain per location? e.g: {basin}.us-east-1.s2.dev where *.us-east-1.s2.dev points to the cell? My best guess is that you prefer to have explicit architecture in case you need to move to a new cell for some reason, but maybe there was a different reason? The stuff around hitting the database on every query that necessitates a firewall seems like a huge risk vs. a wildcard CNAME which adds no additional burden.
Also, unsolicited advice but having your website, integration endpoints, status page and email under the same domain is a bad idea, a single point of failure. Especially a domain that is due to expire in just a few months. You should have a separate domain for infrastructure and a separate status page domain, each renewed for 10 years in advance (the maximum). e.g: s2.dev for website, s2status.com for status page and s2stream(s).com for your Basin URLs.
I wanted to share our journey of running our own DNS server (it's not as scary in the age of AI, but still...). If anyone has questions I can answer them.
If you're running a k8s cluster, you're already running a DNS server. It shouldn't be too scary IMO, but I've also wasted a lot of time writing DNS-related libraries and application in the past, so I'm probably biased.
I find the following part of the article interesting.
These [DNS] records would need strong consistency too. When a basin is created, its address should be instantly resolvable, otherwise clients would be waiting around for a cached NXDOMAIN to expire.
DNS is, almost by design, an eventually consistent system. Strong consistency would mean authoritative servers for a zone have to coordinate updates and agree on the current state before answering queries. That would almost defeat the availability goals that date back to the very origin of DNS. To quote the very old RFC 1034:
Access to information is more critical than instantaneous updates or guarantees of consistency
I am guessing S2 has clients query a specific DNS server directly so that there is strong consistency in practice. Also, it's unclear to me whether it's intended for records to propagate to other name servers. It'd be nice if the article included an appendix for people who don't know what S2 is (or how clients typically access the service).
> DNS is, almost by design, an eventually consistent system.
I agree with this. Offering instant updates creates a large performance benchmark, especially if you experience massive growth. Unless this is mitigated with TTL to some extent? I would personally be looking at how to reduce the database load for misbehaving clients e.g. 5 second cache on all queries or allowing a customer to choose a "tier" where they won't need fast instant updates. If they choose e.g. 5 minutes TTL, you can cache these extensively, even just as disk files rather than RAM.
I'm not familiar with your product, just the blog post + cell architecture documentation. Question: the documentation states that basins are assigned to a cell on creation and cannot move. Why choose to have explicit DNS records instead of wildcards with a subdomain per location? e.g: {basin}.us-east-1.s2.dev where *.us-east-1.s2.dev points to the cell? My best guess is that you prefer to have explicit architecture in case you need to move to a new cell for some reason, but maybe there was a different reason? The stuff around hitting the database on every query that necessitates a firewall seems like a huge risk vs. a wildcard CNAME which adds no additional burden.
Also, unsolicited advice but having your website, integration endpoints, status page and email under the same domain is a bad idea, a single point of failure. Especially a domain that is due to expire in just a few months. You should have a separate domain for infrastructure and a separate status page domain, each renewed for 10 years in advance (the maximum). e.g: s2.dev for website, s2status.com for status page and s2stream(s).com for your Basin URLs.
Out of curiosity, was using PowerDNS with CloudSQL (fed from a spanner stream) considered?
There are always trade offs of course but it would have the advantage of DNS spikes not impacting spanner.
I wanted to share our journey of running our own DNS server (it's not as scary in the age of AI, but still...). If anyone has questions I can answer them.
If you're running a k8s cluster, you're already running a DNS server. It shouldn't be too scary IMO, but I've also wasted a lot of time writing DNS-related libraries and application in the past, so I'm probably biased.
I find the following part of the article interesting.
DNS is, almost by design, an eventually consistent system. Strong consistency would mean authoritative servers for a zone have to coordinate updates and agree on the current state before answering queries. That would almost defeat the availability goals that date back to the very origin of DNS. To quote the very old RFC 1034: I am guessing S2 has clients query a specific DNS server directly so that there is strong consistency in practice. Also, it's unclear to me whether it's intended for records to propagate to other name servers. It'd be nice if the article included an appendix for people who don't know what S2 is (or how clients typically access the service).> DNS is, almost by design, an eventually consistent system.
I agree with this. Offering instant updates creates a large performance benchmark, especially if you experience massive growth. Unless this is mitigated with TTL to some extent? I would personally be looking at how to reduce the database load for misbehaving clients e.g. 5 second cache on all queries or allowing a customer to choose a "tier" where they won't need fast instant updates. If they choose e.g. 5 minutes TTL, you can cache these extensively, even just as disk files rather than RAM.
What is the scale on the graph?
You are. Not me.