John Todd of Quad9

Alright. Welcome back to another episode of Modern Cyber. We've got a topic on deck at least for the beginning of today's conversation that is near, and I won't say dear to everyone's heart, but it is definitely near to the hearts of those of us who have been around this stuff for as long as I have, as long as our guest has, and that is DNS. We are gonna be talking about DNS. We're gonna be talking about some of the modern, let's call it the state of the DNS, how has DNS evolved or not evolved as the case may be.

And to join me for this conversation today, I'm delighted to be joined by John Todd. John Todd is the general manager for Quad9. He's been involved in Internet infrastructure, networking operations, and DNS related roles for thirty five years as an engineer, manager, and IP enthusiast. And John is currently the general manager at Quad9. John, thank you so much for taking the time to join us today.

Thanks for having me here. Well, let's get into it. I I said in the intro that DNS is kind of near, but not necessarily dear to my heart. And I only say that because it it's the kind of system that we're all using, you know, a gazillion times a day. It's been around for thirty, forty, I don't know, fifty years.

You you can give us the history on that. It's the kind of thing we take for granted. We don't know that it's there. The Internet doesn't really work without it, at least from a practical level. Nobody wants to go into straight IP addressing.

But I guess one of the first questions I have is, first of all, has DNS evolved? And maybe, I don't know, maybe just give us a quick overview, a quick history of DNS over the last decades. Last decade. So I'm gonna skip the most early parts of the DNS, and I'll just mention that, like, the Etsy host file, right, where people would actually move files around containing all the names that mapped all the IP addresses of things. So we moved away from that.

We moved to DNS, which is a distributed architecture, and you're probably familiar with, you know, everyone who's got a domain name has an authoritative name server operator. They contain Yep. Yep. They have essentially the the canonical phone book of what the names are that map to addresses or services. Using addresses is becoming a little bit out of out of vogue.

It's actually services. The DNS is doing more than just name to IP address mapping these days, and I'll get to that in a second. But so and, of course, then every device on the Internet, almost almost, I'll say, not not every transaction, but 99.99% of all transactions start out with a DNS request. Like, where do I I wanna connect to the service, and how do I get there? And everybody has then to serve those answers a recursive, DNS system that actually takes that request from a client and then performs some lookups and says, alright.

Alright. We're gonna start asking questions about how do we get this namespace. Where do I ask the questions? What server has the answers to these names? And there's a very long process of recursively asking different parts of the namespace where the answer lives.

And it is it is as it occupies a strange place actually in the Internet because it's not a network service. It's not TCPIP. But yet it is so fundamental to every transaction on the Internet that it might as well be part of the infrastructure. And that's how we're trying to kind of structure things, like what like, can make DNS considered as part of the infrastructure from both a technical perspective and then also from a political perspective. We'll talk about that in a second too.

Okay. The, as each client does these requests, right for the last, you know, 30, it's been relatively generic. The the client makes a request. It gets a response back, and it makes the connection. In the last really in the last fifteen years or more, there's been the concept of actually doing some more stuff in the DNS.

Yeah. And and, one of those things has been to kind of put a security layer in the DNS, and that's actually been quite effective. Meaning, you ask your recursive DNS resolver a name. And let's say it knows that the the recursive resolver has a list of bad sites in the Internet, meaning, they're trying to distribute malware or they're trying to be phishing sites or whatever. They're they're trying to do some harm to the end user.

And so the DNS resolver will then either give you back a negative answer, meaning saying, I don't know where this is, or in some cases, it might give you the wrong answer and say, no. You should go to this website instead. So those functions have been around for a while. And in a very high level, that's what Quad9 does. So Okay.

There's a second thing that Quad9 does as well as protect you from malicious sites, and that is that we actually also we don't give anyone your data. So this is kinda the core fundamentals about what Quad9 does. We actually we do recursive DNS, but we also have a security component. But then we also don't collect information on end users, and that's actually very different than maybe some of the other services like your ISP. Do you really know what your ISP is doing with your DNS data?

So many of the changes that we're seeing in the DNS are not just in some of the technical measures, like encryption in the last seven or eight years has become a thing. We're now encrypting DNS as one of the last protocols to get encrypted, which is great. Yeah. So now there are at least three different major ways of encrypting, if not four major ways of encrypting the DNS. That's fantastic.

We support many of those. But now the question of, like, what is being done with my DNS data is becoming something that people are really interested in. Because if someone sees your DNS data, they might as well be looking exactly at the web pages you're looking because they know based on the names you're looking at where you're going. So, we're trying to kind of push the DNS in a couple of ways from a technical perspective. Quad9 is trying to as, again, doing things like we are the first resolver to support standards based encryption, and we are the first resolver to really have an incredibly strong privacy policy.

So our organization is based in Switzerland as well, kinda put our money where our mouth is, where if we don't adhere to the privacy policy, there are actually criminal, charges that get brought against us. So that's, that's actually that was a reason for going to Switzerland because we have to comply with those privacy issues. And we're really trying to push that ethos into other people's thinking. Like, do you should be asking what your ISP does with your DNS. And if you don't like it, you can go to Quad9.

We're not trying to control the world. Right? We're a nonprofit, but we're really interested in getting everyone to have better delivery of DNS data because of privacy, because of security components. So the changes that have happened, you know, as as I said, encryption has happened, to a a relatively standardized protocol that's really been untouched for a very long time. And around 2017, '20 '18, encryption started to happen, and now there are several different significant encryption protocols.

Privacy has become an issue. People are really interested in, okay, what are you doing with my DNS data? Yep. And who's collecting it and who sees it and why? And then, you know, performance improvements and kind of additions incremental additions onto the DNS protocol itself are starting to happen, which are interesting, but far removed from, I would say, the end user's view, but they are interesting nonetheless.

Some things like discovery of encryption protocols and discovery now of services, and there's some new things that are coming up from the IETF, which have some real interesting promise to them. Awesome. Awesome. Thanks for that. That's a great overview and a great summary and introduction.

There there's a few things I wanna dive in there to make sure I understand them correctly. Number one is you you said something there that I wanna make sure that is crystal clear, and our audience has a lot of, you know, pretty technical people. So you mentioned that, you know, DNS is not TCPIP, but my understanding is that DNS kind of runs over TCPIP. Is that kind of fundamentally incorrect? Is there a DNS without TCPIP?

No. There is no DNS without TCPIP. Okay. DNS has traditionally been a UDP protocol. It's actually one of the few protocols that uses UDP, meaning it's not guaranteed.

Right. But, that's changing. Again, all the well, not all. Most of the encrypted protocols use TCP now by default, which means that you get the full transaction. You get guaranteed delivery.

You've got the handshake. You've got the kind of, you know, stronger connection. Right. But if we look at our network, if we look at the number of queries, that we do that we deliver per day, still the majority the vast majority of them are UDP, which has pluses and minuses from a security and reliability perspective. So we're really trying to encourage encryption as the as the way people should go.

Okay. And, I mean, it's been a long time since I've been hands on a firewall. Thank goodness. I I don't miss those days at all, but I always remember, you know, kind of, you know, DNS port 53, and UDP protocol presumably, and it was one of those services that you kind of always had to leave open to to, you know, to zero dot zero, right, and and make it publicly available. People didn't really get to DNS.

Yep. Yeah. And so there's a couple things that I also wanna understand. So you mentioned that, you know, kind of DNS has evolved into being more kind of, let's say, serving services or serving lookups for services as opposed to IP addresses. What's the distinction there?

Because when I think about services in a lot of context, I still I understand that there is, let's say, an API that I'm calling or there is a third party external component that I'm calling, and that is a service or let's say within a Kubernetes cluster or something like that, that's a service. But I still tend to kind of logically equate in my mind that that is a DNS lookup to an IP address. It might be temporary or it might be like local to that, Kubernetes cluster or something like that. What's the distinction that I'm kinda missing there between the service and an IP lookup there? So, it does both.

It actually still does IP lookups. At the very end of the process, you're still doing a name to an IP address mapping. But, to give you some ideas of what else the DNS can be used for, and this has actually been around for quite some time. There's there's what's called an SRV record or service record. And that simply says, you know, for this for this domain, let's say, for example, .com, I'd like to know what the mail servers are, for example, .com.

And you can ask the DNS about a mail service record, and it'll spit it'll send back to you a list of mail servers. Classic MX records. Well, but but it it's more than that. So let's say you don't have web servers. Let's say you want, I'm sorry, mail servers.

Let's say you want web servers. Or let's say you want, SIP servers, like voice over IP servers. So SRV records can contain any of the standard service names that you're sort of familiar with, like SIP or, mail or web or, you know, any basically, any Internet service. You can actually embed that into the DNS. Frankly, that hasn't been a really popular protocol to use.

Although, it is interesting to see some of the Kubernetes stuff that is starting to push service records back into the DNS because of the concept of automated discovery of services. Yeah. So, but but SRV records are something that I I think has been underutilized in the DNS, and we do see some now some more use of it. But, basically, it says that that for a particular zone, a particular name, there may be services associated with it that you don't even know the name to, but you're you're just interested in, like, what is tell me what the services are associated with example.com. And you can get a list of those back with SRV records.

There are other oh, just go ahead. But that that would be something like if I'm doing, let's say, an NS query type of thing and I, set my query type to service and then I just kind of request a full dump of all the service records for that domain? You some name servers will respond and give you all of them at once. That's called an any query, and that's the hopefully, those are becoming fewer servers that are gonna respond to those. But you I actually think there might be more, but we'll come to that in just a second.

Well, there's there's what's called an any query, and that one basically says, show me everything you can see for this zone. And those are sort of dangerous, frankly, because used by every scraper on the web and every website pretty much every website and every service online is actually getting scraped more and more and also getting kind of cataloged and indexed and used by AI engines for training purposes and so on. So hence my my theory that actually the volume of these queries may be going up. You're right. The volume of the queries may be going up, but we try as a it's a it's a mild preference that people do not actually enable any queries on their systems.

Okay. The reason for that is the reason for that is any queries basically say, show me all the records for a particular zone. And that's actually a really big blob of information that comes back. And because the DNS is still primarily UDP based, that means you can send a very small query to an authoritative name server. And you get a really big query or really really big response in return via UDP.

Yep. That allows spoofers to say, oh, I'm gonna spoof a thousand different IP addresses, and I'm gonna send the same question to this to the authoritative server, and I'm gonna pick a target that isn't me. And so you get a multiplier. Like, you send you send a thousand any queries at, you know, 500 bytes piece or 200 bytes or even less or 50 bytes to a name server, and then it responds back with three k each in response. X thousands.

Yeah. So you're basically any any queries are an amplifier for attack vectors. And so we're Yeah. The so called DNS, denial of service attack. It is a method that you can start to use for that.

Yes. And then, you know, authoritative operators should be limiting the number of any queries they respond to, and recursive operators should be limiting the number of queries that they get. There's there's all kinds of different ways to defend against that. But, going back to the concept of services in in the DNS, I think that that's actually a really interesting model because people aren't are starting to become less and less aware of even, you know, w w w as a prefix. People aren't even typing that.

They're just putting in a domain name and then Domain name. Yeah. Expecting whatever it is that they're trying to get to, whether that's a video call or whether it's a web request or whatever, that that somehow automatically there's gonna be discovery happening. So I'm encouraged to see some of that happening. But I I mean, even looking at like, if you look if you have example.com or, actually, we use a case study, we can actually look at google.com or apple.com and look for TXT records associated with apple.com.

You're gonna get back a huge list. Yeah. Because everyone puts in, you know, their identifiers for particular services as t s t records. Of ownership of this or that or yeah. The the DNS is getting all kinds of stuff crammed into it, and that's not gonna stop.

I mean, if Yeah. If past behaviors are any indicator of future future results. Yeah. Yeah. So the DNS has been getting more and more loaded with things.

So it's becoming more and more critical that there is trust in the DNS, that the answers that you're getting back are in fact true, that they're not being intercepted, spoofed, blocked by somebody else. And then that the answers you're getting are confidential, meaning that no one else can see what you're doing. Yeah. And so there's two really widely different, protocols that are being used for that. One is DNSSEC, which you may have heard of.

DNSSEC basically says that the answers you're getting are in fact the correct answers, and there's a cryptographically signed component that says, yeah, that what you're what you're looking at is true. And then encryption protocols like DNS over TLS and DNS over HTTPS give you the guaranteed, ability to say that my the transaction that I have with my recursive resolver is not visible to somebody else. Right? So and there, again, the double edged sword from a security perspective, if you're encrypting your transactions to your to a recursive resolver that is not in your network, like Quad9 as an example, 9.9.9.nine, your security people for your network can't actually see what you're requesting. Right.

So that's a downside from a security perspective. Now From an enterprise Right. From an enterprise say from an organizational perspective, I don't know what the people in my company are are looking up and getting. Yeah. Right.

So but on the upside, if it's not your company that's looking at, let's say it's your government looking at it and trying to create a profile to prevent you from going to something that, you might want to go to, whatever that happens to be, then perhaps that's an upside for you. So Yeah. Again, there's the the question of security or or anonymity slash encryption is a double edged sword in all cases, and we're having to walk that line, well, with every transaction that we do depending on where the where the query comes from. Got it. Got it.

So you talked you mentioned a couple record types there that I wanna dig into a little bit because, you know, the services record, for instance, the example that you gave, I guess, I think a lot of people, myself included, we would have historically said, oh, yeah. You just use a records and c names for those things. But I I get the point that you're making around like, hey. The service thing might be a little bit better because it's a little bit more descriptive. And is it more descriptive because it's a different format of data in the record where, like, you know, a's are very, specific.

They expect an IP address. And c names, again, they expect, you know, kind of an FQDN, in the in the, data field of that. Is there, like, more flexibility with the service record, or is it what's different about it? Quite quite a bit. Service records as I said, so I'm going down this this path.

I'll say that service records are not as well utilized as I'd like to see them being. So Right. But this is the way they do work. Service records have the ability to be number one, they can be changed by the operator of the zone very quickly. So you can you can make modifications quickly, and people don't have to remember what name that they need to go to.

They just simply put in, alright. I'm looking for the the voice over IP server for enter for example.com. And then the operator of example.com has the ability to automatically change that rapidly as they wish. So there's one flexibility. Second flexibility is that you can put multiple records in there.

You can say, I have preferences. Here's the first one you should go to. Here's the second one you should go to. Here's the third one you should go to. Okay.

So it has it doesn't have just one answer. It actually can give you multiples, and it can give you them in an order that you'd that the operator of the service would like you to try them in. And can that be different for different resolvers or different geolocations? It can be different for different clients. So depending on where the client's coming from, whether it's coming from a different recursive resolver IP address or whether it's coming from some a different geography, yes, you can give different answers back.

So there's so those SRV records are sort of flexible. The third thing is the SRV records also include things like ports. So you can actually tell people, okay. I don't wanna just use the standard port. I want you to just use an alternate port.

So it is it is a fairly flexible model. I would love to see it used more, and, it is used in some environments, as I said, like, there's there are components of that in the voice over IP world that are used extensively. There is some use of that in in the the large scale computing world. I I'm interested in seeing that transmit further down the stack. Got it.

Got it. And so when we think about one of the other record types that we mentioned there or that we discussed is kind of the text record. Yep. And I see the text record all the time. You know, pretty much like every Catch the same.

New SaaS service, every new SaaS offering I sign up for, it's like, oh, do you want us to be able to send email as your domain? Then you need to prove domain ownership, and you do it through this text record. And, oh, do you wanna, you know, verify your identity? Awesome. We'll do it through this text record.

That's the primary use case that I see for a lot of these things. What are some of the other ones that people should be aware of? Well, I'll I'll say, first of all, an opinion piece here. Like, I'm really opposed to putting all those TXT records in the top of the zone. There you can put you can put, service, names in there.

So you can put underscore and then an a servicename.example.com. Okay. Use that as your reference because that way you're not polluting the top of your Apex with all these different records and making those response to some giant lump of 3,000 byte responses. So create discrete records. If you're if you operate a service, please create a discrete record for that.

So other other records that we're seeing, so, boy, there are a bunch. There are now, some things for discovery. There's, allowing basically, you say, okay. I wanna I wanna know what the recursive resolver is for my usage. So there's what's called dynamic discovery of resolvers.

Okay. So there's there are records now for that. There are SRV records, TXT. I I'll say that, unfortunately, TXT still still seems to be the kitchen sink. Everyone still seems to throw things in the TXT.

And it's totally unformed to anything. It's just a generic key value pair. I by the way, I've seen people using them almost as a, you know, as a mini relational or Yeah. Key value database for for application lookups. Yes.

There was someone I think that even did the Star Wars opening scroll as TXT records, so you do a recursive fetch for that. Yeah. That's there's all kinds of things shoved in the DNS. You know, the the and the again, sort of like not in favor of TXT records. A lot of people use TXT records as exfiltration.

So, you can you can actually, you can create a tunnel over DNS. You can create a TCPIP tunnel over DNS with a specifically configured authoritative server and a client software package. So, basically, by sending packets in and encoding them into requests and responses that are TXT record models, you can essentially create a communications channel for exfiltration of data in environments that are really tight. So, it's a constant cat and mouse. You know, How do we we we actually don't allow that through our network, because it's an abuse of the DNS, essentially.

It's abuse of recursive and authoritative services. So, you know, how do we slow that down, stall it, you know, prevent that from occurring? That's there's a lot of nuances now. And the recursive operators recursive resolvers used to be something you could really quickly set up. You'd set up bind.

You'd say, okay. You're now you're now a recursive system. Away you go, and you'd forget about it. And that sort of isn't the case anymore. You need to have some reasonable expertise knowing what you're doing to set up a a recursive system for your enterprise.

By the way, there are plenty of great open source solutions out there which do all this for you now. Yeah. Yeah. But, you know, really doing it well and at scale and in a private way is is becoming more challenging for better or worse with all these new new extensions to the DNS. So, services like Quad9 exist for people who don't wanna roll their own.

Yeah. And so, we're happy to take that. But we're also happy to see people using it themselves and figuring out how the DNS protocol works and and doing their own deployments as well. As long as they can do that securely and safely, that's we're all for that. Yeah.

Yeah. Securely and safely, I think, is one of the key things. And I would say one of the other things around it, I mean, whether it's the security or the safety aspect or it's just actually the accuracy. Because I find that, you know, for many roll your own DNS implementations that I've seen over the years and bind is probably a trigger word for some people in our audience as it is for me, you know, that we've all made that mistake of, oh, I forgot to update the serial number on the file or oops, I missed one final, dot at the end of a line of a c name record and everything after that is just completely borked. So, you know, security, safety, yes, awesome.

That is a big point of the podcast, etcetera. But, you know, the accuracy and not borking your own domain is is definitely a strong argument for not doing it yourself anymore. On the authoritative side, definitely. On the recursive side, you have run a little lower risk for shooting yourself in the foot. Yeah.

However, there is still a risk and that we see this quite frequently. People accidentally they'll they'll they'll make they'll roll their own recursive resolver, and they'll get that running or And then what or recursive server. And what will happen is they will leave it open to the world. And so everybody can use that. And so suddenly, you get scanned, you know, of course, as everyone gets scanned every couple minutes now.

You get scanned, they say that there's an open resolver, and you'll start to see queries coming from all over the world using you for not necessarily the right purposes, right, for denial of service attacks or Yeah. You know, just generally things you don't understand. So there are still some ways you can shoot yourself in the foot, doing your own recursive resolver, but, it's a little less risky than running your running, authoritative on your own. But Yeah. Yeah.

Certainly. Still got lots of lots of speed bumps now if you wanna do it at with the current technology. Yeah. So speaking of recursive DNS, you said that you can think about using recursive DNS as a cybersecurity tool. What do you mean by that?

So, the as I sort of said at the beginning, that there there is the ability now to include, block lists into the DNS. Meaning, if you try to look up something that's malicious, you can actually embed lists of bad sites into your DNS recursive system so that when you try to look up or when all of your clients try to look something up, they get essentially a no answer. They can't connect. Yeah. This is really important with IoT devices and things where you can't actually install any software.

There's no way for you to get in there and, like, figure out how you're gonna block bad things from happening. However, everything uses the DNS. So it's really that's the only other than TCPIP, it's usually the only API that you have to even hope to get some kind of security embedded into devices that you don't control. So Quad9, like, we are we are one of those services that embeds, malicious host names into our feed so that we block them for our end users. Yeah.

We get that we get that from around 30 different open and and as well as commercial origins for recursive or for sorry, for malicious names. And that actually works out really, really well. Both sides of that equation benefit from that. Mhmm. End user benefits because they don't get connected to to phishing sites or malware sites.

Right. And the threat intelligence providers that we have benefit because we give them some volumetric insights, like how many times did this domain get queried. So they can understand a little bit better about what exactly their threat feed does. So it's, it works out for both sides. Well well, two kind of follow-up questions there.

Number one is, that's awesome. That makes a ton of sense. That also sounds like a continuous game of whack a mole. And Yes. I I'm kind of curious how you manage that.

And number two is you talked about working with some external providers on the threat intelligence side. Talk to us a little bit about that. Where where does this data come from? What kind of data is it? Sure.

So, yeah, it is a it is a game of whack a mole. And it's a game of whack a mole that's pay played at, you know, second by second, speeds. Literally Internet speed. Right? Yeah.

So we have about, I think, sort of between three and four million different domains that we keep in the list at any one time. But those rotate in and out very rapidly, and that's an automated process, from our threat intelligence providers. They give us these lists and update them. So things appear really rapidly. Like, new domains are registered and used for malicious purposes within minutes of their being registered.

So our job is to figure out how to get those how to have our threat intelligence providers note, note that those exist and then feed validate that that they are, in fact, threats and then feed them to us as quickly as possible so we can block them. So we we ingest these these lists. It's not exactly nothing nothing is truly real time. Right? But we ingest these lists very quickly from our threat intelligence providers and then embed that into the feed, distribute it out.

We have, right now, 300 and some odd sites worldwide where we actually have systems deployed. Didn't really talk about that at all, but, like, recursive resolution needs to be fast. And so in order to be fast, it needs to be really nearby to the end users. So we have even a speed of light and geolocation Yeah. Kind of, you know, proximity challenge that you're trying to overcome.

Yep. Yep. And so we use Anycast, meaning that we advertise the same address, the the same slash 20 fours or slash 40 eights in every location where we have equipment. And so people sort of automatically go to, I won't say the best, but from a networking perspective, the shortest path is the one that they take. Right.

And so you terms of, like, millisecond resolve time or No. Or hops? Or As far as the as far as BGP is concerned. Okay. And so that you could sort of say that's number of networks between us and them.

Right. But in some cases, it's actually manual. Sometimes people will manually try to prefer traffic to come to us in certain certain ways or not. Usually, it's usually, that's tightly integrated with milliseconds. Usually, it's tightly integrated with delay, but not always.

And there are always weird cases. Firsthand through that, by the way. I didn't yeah. So the the goal is if you get more and more and more sites out there, if you distribute more of your network and have exactly the same prefixes announced from every one of those locations, overall, people will go to the fastest location because it's gonna be close to them network wise. Yeah.

And so our you know, that and and the old saying around the the the the Motor Head community, there's no substitute for cubic inches. Well, there's no substitute for number of any cast deployments. Like, the more you can have, the the more you're gonna be able to get end users to the fastest one. So, the cybersecurity needs to be fast, it needs to be private, and then it needs to be good. So we have about 30 providers, as I said.

We we get from them the list. We give back to them sort of real time information about how many queries they're getting for those threat domains. They, in turn, will repackage or use that information to actually sell to their clients because we get, you know, their organizations, like, like, as an example, you go to our website, you'll see IBM. We have actually a big list of our partners on the website. But IBM, as an example, repackages the information they get with us because they discover they they give us a domain.

Within a few seconds, we start to give them insight into how many queries this domain's getting that we're blocking. And then they say, this is actually really this is advancing in Asia. And so they take that data and through whatever mechanisms they have, which I have no insight into, but they they actually monetize that by creating different levels of service they can give to their customers at at a much higher level of integration with different SIEM platforms and and things like that. So Mhmm. The the data that we provide is essentially first insights into the way that domains are operating on the Internet from a threat perspective.

And that in turn helps our threat intelligence providers, but first of all, it helps our end users because they're getting blocks, really at the head of the curve. Interesting. Interesting. It you know, I have to ask the question. You mentioned at the beginning that Quad9 is a nonprofit organization, but some of that sounds like it's gotta be kind of expensive.

It is. First of all, there's all the servers that you gotta stand up. And second, if you think about the I I don't know what the policy is around data handling, but at least just processing the data and responding to all the queries, whether you store that or not, that's a fair bit of bandwidth and a fair bit of compute power. It is. How how is the organization set up, and what's the kind of the corporate structure and and the business model if there is one?

So QuadLine is a nonprofit. As I said, we're based in Switzerland. We are, we're a small team. We're, you know, 10 or less full time folks. But we bat well above we we box well above our weight, as they say.

So what we do is that that we're we are extremely fortunate that have a number of different partners who give us most of our facilities. Pretty much every location where Quad9 exists in the world is provided for free by different partners. And some of those partners are give us many dozens or even hundreds of locations. Our, p c PCH Packet Clearing House is our biggest partner. But we have other partners who give us specific locations like, there's a company called Edge Uno which is in Latin in South America, which is great.

They give us facilities. There's I three d which is Europe and some of The Americas. There's Equinix, who gives us a footprint all over the place. So they they believe in the mission. They they like what we're doing.

We're making the Internet a safer place. We're making the Internet a more trusted network for their end users. So they they like what we're doing. So we get a much bigger footprint than we're paying for because we're not paying for it. We're getting it for it at at no cost, essentially.

So the biggest item in our in our in our budget is actually is a zero balance for us. So Okay. We still have to ship equipment out to those places, though. Yeah. And so that's a big that's a big cost and a big effort, and that's actually where a lot of our time goes is actually figuring out where we're gonna deploy, shipping equipment, managing customs, getting coordination with the local site contacts to get things turned off.

So there's a lot of work there when you're looking at dozens and then hundreds of locations to ship things. Got it. So our costs primarily are staffing Yeah. Equipment, and then what I'll call overhead, so legal costs and and other things. So we're we're extremely efficient with our capital.

We don't have we don't have a an office. We don't have a physical instantiation. Everyone is remote. We are extremely, cost efficient with buying things. All of our equipment is is off the used market.

But the good news is that companies go through things on a four or five year cycle, so it's actually still fairly new gear. Yep. And so, how do we make money? Like, how do we do this? So we get grants.

There are a number of different organizations who give us grants. So we've got grants from a bunch of different places like the EU, like OTF in The United States. The Craig Newmark Philanthropies gave us some money last year, which is the announcement on our website. So there are people who believe in what we're doing, and and they understand what we're doing makes everyone safer. And it makes everyone more able to trust the Internet as a as a basis for learning, as a basis for economic use, as a basis for, just the day to day functioning of the world now, which the Internet has become.

Yeah. We also have partnerships with some commercial providers, like, as an example, IBM. IBM is one of the few big organizations that really understands the benefit that they get from this relationship we have with them for threat intelligence, and they're willing to give us some sponsorship for that. So we make money through a whole bunch of different ways, primarily through people who believe in the mission, but but also in some cases through people who both believe in the mission and get some benefit out of of our existence and the fact that they can they can then also improve their product with some of the information that we give back. So so I'm curious then cause there's part of me that says that wants to tell everybody, hey, just immediately go change your, primary DNS, lookup servers to 9.9.nine.

But then on the other side of that, it's like, hey, well, that actually incurs more cost for you guys. Yeah. I I again, a saying I use is we lose money on every user, but we make it up in volume. Yeah. Exactly.

So, no, we definitely encourage more people to use the platform. It is a challenge for us, but we get grants and sponsorship by continuing to show that there's value. And the way that we, the way that we show value is that there's a bunch of others. But the the first one is that we we continue to grow at around 2% per week. Yeah.

Yeah. Which if you really think about that number, 2% per week is a terrifying figure. Yeah. We're we're we we're relatively confident we're well above a hundred million users now Okay. Worldwide.

We have an extremely strong footprint in places where no commercial organizations go. Africa, we're the largest. I we're I I relatively certain we're the largest and certainly most distributed recursive system in Africa. But we go to countries and islands and places where there is no real commercial benefit to go because our goal is not commercial. Our goal is not to somehow extract money from people in some way.

Our goal is to make people safer and Yeah. To have them use the Internet more effectively. And so, that's that's how we show our value is that growth. To to get back to a previous question you asked, which I which I didn't answer, which is, okay, what do you do about all the data and the volume? Yeah.

There's actually a really easy answer to that, which really stymies, people who try to ask us questions for for, audit purposes, and that is we don't collect it. Yep. We don't collect data. Meaning, the p I PII, personally identifiable information, IP addresses in our case, We never collect that information. We actually discard it.

The minute that we answer the question, the IP address goes away. Yeah. So we have avoided an enormous amount of expense and complexity and compliance rules by simply never collecting any PII on users. But you do collect the number of queries against particular records or domains. Yep.

Yes. We do. Okay. But we but we aggregate that. So we keep a counter.

Right? It's just cheap to keep a counter. Internet, automatic counter. Yeah. I totally get it.

And so, like, you don't know that Jeremy looked it up, but somebody looked up example.com, and they maybe looked up, you know, 10 times in a five minute period or whatever the case may be. So the block records, we really keep close tabs on. And, actually, if you go to our privacy policy, which we're quite proud of our privacy policy, it took us a really long time to write it. And it's actually been used as part of an RFC as sort of an example. But if you go to our privacy policy, you'll see all the things we collect and and how we collect them.

And then, like, again, counters are the biggest way to deal with things. Like, we don't actually store anything about the end user, except for some really rolled up generics like what country, you know, how this country made this many block queries to this domain. So it's really hard to go backwards and try to figure out anything about individual users. In fact, it's it's effectively impossible. Yeah.

So but there is still a lot of work that has to be done in collecting those counters and, you know, creating the tooling to to do that at scale. And in fact, we've been very active in writing some public open source tooling to do just that, to to add additional features to various platforms, so that DNS is an easier protocol to start to understand at scale or even even in a small scale. We've we've helped with some tools for that. Awesome. Awesome.

Well, we're coming close to the end of time here, and I've got a couple of questions that I wanna kinda finish up on. There's one that's been kind of rattling around my mind, and I've seen it a couple times over the last couple years, and you may have as well. And it's every now and then, you'll see one of these statements that, you know, we're using the same Internet as thirty years ago, twenty years ago, whatever. And, you know, people who will come out and say like, oh, okay. You know, I p v four is kind of on its last legs, whatever.

I I don't think that's necessarily the case. I think we're gonna be using it probably until the end of my career. But then the one that really for this conversation jumps out at me is I've seen people making a claim, oh, DNS is broken, and it has been for ten years and so on. And I've never really understood, like, what's behind that argument. And I'm curious, a, have you seen that argument?

And if so, have you heard any of what the actual foundations of that argument are? Well, okay. There's two I'm gonna speculate on when you say the DNS is broken. There's two camps that I've heard of, and maybe there I'm I'm certain there are many more. So Yeah.

Forgive me if I'm not complete. The first camp, when people say the DNS is broken are people who have a com a commercial alternative that they'd like you to use. Okay. Like what? I mean, what I I can't even understand what an alternative to DNS would be.

There there are some efforts to include blockchain in the DNS and basically distribute, the the DNS in some, verifiable method using blockchain technology. Okay. I think that's interesting, but I have not seen an offer yet or a protocol model that seems to make sense or that scales out. Yeah. Yeah.

So but and usually there's and usually people who have an offer like that, they're there's they're like, and by the way, my company does a really good job at that. And so there's, like, a there's a self serving aspect to it, and people immediately don't trust that. So I'm gonna say the first people that say the the DNS is broken are people that are trying to come up with something that either improves their business plan or is their business plan in some fashion. The second is that there have been a number of people who have said that DNSSEC is broken over the last fifteen years or so. And that that has a little bit more validity to it.

If I was having this conversation eight years ago, I would say, alright. You maybe you're maybe that's true. But, DNSSEC is a method to validate and make sure that the answers you're getting are truthful. That is sort of done now. Like, I don't Okay.

We we offer so just to give you some background, like, eight years ago, there were some effort oh, actually, more than eight years ago now. There were some efforts to basically force DNSSEC, meaning force this this cryptographically assured chain of of ensuring answers are correct, down the the throat of the the US government. And, like, every US government agency had to use DNSSEC, and that was kind of a disaster. It it didn't get well done. The the technology was still relatively new, and it was roundly disapproved by a large number of people.

And so there was some there's still some hurt feelings over that. Fast forward to 2025, Quad '9 now and actually has since its start eight years ago, used DNSSEC as a default. Meaning, if there's a DNSSEC fault, we will not answer the question. So if DNSSEC was broken, we would be hearing about it at large scale from millions, tens of millions of end users would tell us, hey. This isn't working.

DNSSEC does work. It's still not as widely deployed as we would like. But it Yeah. As far as being a problem, our opinion right now is that DNSSEC is stable and robust, and there may still be some nits to work out, but it's it it does the right thing, and it helps people answer questions. So those are the two things that I can come up with where people are saying the DNS is broken.

The DNS isn't broken. It's working exceptionally well. It has. And since we talked about, everybody who's listening to this podcast uses it a gazillion times a day. Yes.

They do. You know, with the exception of an occasional airport lounge or hotel Wi Fi Fi or, you know, coffee shop Wi Fi where occasionally DNS will fail for, like, a minute and you end up resetting your Wi Fi connection, I've rarely run into cases. I I've seen the odd example of being in one of these, you know, public Wi Fi systems or something where, like, only Google resolves, for instance, and you're only able to issue Google queries. And Well, interesting you brought that up. That what you're seeing right there is actually one of the things that we're that personally I would like to see die in a fire.

And that is because what's happening in those Wi Fi and airports and and everything, they're actually intercepting your DNS queries. The the network somewhere is intercepting your DNS query and rewriting them. That's terrifying. That is that is imperceptible from an attack against you. But they're doing it, you know, for your, you know, for your good.

Right? They're trying to send you to a splash page or something. But that is that is super broken. Right? And you're just accepting that.

You're saying, oh, okay. So it wants to hijack my DNS request and send me somewhere other than where I was intending to go. Yeah. Okay. Let's do that.

Like, that's that's a bad, bad model. And so I'm I'm interested to see some of the new protocols again that are coming out for allowing that kind of thing to happen without breaking without intercepting or injecting, false answers into the DNS. So I'm Yeah. I'm hopeful that in the next couple of years, we'll see improvement there. But those that that problem you just described is a result of the DNS being an unencrypted Right.

And unsecured protocol for so long that people have just assumed that it you can break it whenever you'd like and no one cares. And that really should not be the case going forward if we wanna have a trusted Internet. And as far as kind of workarounds for that, you know, stuff that I've done in the past, number one, turn on a VPN. And as long as you can resolve the VPN provider, then, hey. You know, you're you're no longer facing that problem.

Number two is, you know, just manually change my, my DNS servers. Right? And and kind of say, like, I don't actually care as long as I've got a trace route that can get me to, you know, nine dot nine dot nine dot nine, then, you know, I'll just send all my queries there instead and tack with you in your intercepted DNS queries. Yeah. Yeah.

That's that's one way to solve it. Some of them won't let you out until you somehow go through their splash page. That's another problem. Yeah. So Yeah.

But that's but that starts to move into the operating system layer where the operating system has to figure out, okay. I've just woken up. And what do I do when I first do know? Gotta negotiate the roundtable and everything associated. And I have to figure out what the DNS server is, and do I trust the DNS server?

Am I being intercepted? Then there's, again, there's work being done right now in the ITF that to sort of to smooth that out a little bit better because it has been a problem. And that that particular case you're describing is an issue, but I would say it's an artifact of this being an unsecure protocol for, you know, thirty plus years now, forty years. Yeah. Yeah.

Awesome. We will include the DNS haiku in the show notes, a link to the DNS haiku. I am sure you are familiar with it, John. It's it's always the DNS? It's always DNS.

But, you know, I've spent the number of nights that I've spent troubleshooting problems, the number of and and I think many of our audience will be able to relate to this. But I wanna close out on one question that was something that you kinda mentioned towards the beginning of there, which is that, you know, DNS has kind of gotten to almost a political level and kind of a I think you called it layer eight level. So what do you mean by that, and how should we think about it? So it's a particularly interesting time to be in the DNS, and I say that with the you know, may you live in interesting times kind of curse model. Yeah.

Yeah. So as a recursive operator, we have systems all over the world in, you know, more than a hundred countries. And the DNS is becoming the last place where people can put their hands on the throat of the Internet, meaning that they can block content. If you block people's ability to know that a site exists, you are effectively blocking the ability for that site to deliver content. Right.

So because that's the case, then now there are governments and now there are policymakers who are looking at and applying laws to d n recursive DNS and saying you cannot it is it is not allowed to recursively resolve these names in France, as an example. And we have a current lawsuit in France where, Canal plus has decided that they're going to restrict us from resolving a certain set of names in France for what they claim are copyright reasons. We are very much in favor of copyright holders being able to defend their rights. That's not the point. The point is that we're creating a system which allows we're we're creating architectures which allow governments which allow, maybe not even governments or basically just pay policy makers of any kind to tell other organizations maybe not even in their own country, what you are allowed to resolve and what you're not allowed to resolve.

That's fundamentally extremely troubling because it essentially attacks the infrastructure of the DNS at one of its most core levels. It's not even you're not even talking about the content provider. You're talking about somebody else, a third party, which was what Quad9 and other recursive resolvers are. You're saying, I'm not since I can't go after the third party or I don't want to go after sorry. I can't go after the content provider and I don't want to go after the content provider.

I'm just gonna go after the people who've resolved the names. That is really, really problematic Yeah. And is gonna lead to really bad outcomes. Like, as an example, in France, right now, there are a number of websites that that we had to block because of legal demands by Yeah. Canal plus and the French government.

We have to block those worldwide because we don't know who a French citizen is. Like, who is who is French? Yeah. Yeah. Like, okay.

Well, no. You could argue, well, if it comes from this IP address, it's French. No. It isn't. Yeah.

Yeah. There is no canonical mapping of IP address to citizenship or jurisdiction. No. And what about the islands? What about the the islands off The United States?

What about New Caledonia? What about all these other exceptions? What about French citizens who aren't even in France? What if they're traveling? Are they allowed to view these sites?

Yeah. Like, the complexity of that answer becomes unmanageable if you try to apply these laws. So Quad9 has actually found itself in this place before. We actually had a case in Germany, in 2021 where we the German, Sony, entertainment in Germany told us to block certain sites, and we went through two years of lawsuits and and appeals in Germany until we won that. We won that case in Germany.

And it was such a a a an assured win that the final court said, not only did you win, but Sony can't appeal. It was that cut and dried. But but entertainment organizations and rights holders who wanna control the Internet are gonna try this in country after country after country. So that's where we're talking about the DNS now becoming a political problem Yeah. As well as a technical one.

And so we're having we're having this both solve the user growth and all these other problems in the at the protocol layer. But now we're being attacked because of organizations that aren't even us. They're, like, very, very distant from us, but we're we're the ones who have to bear the brunt and the cost of of doing these actions. So we're gonna fight that and, I'll put in a quick plug. Please donate to Quad9.

We'll put your money towards good use. Go to the website. There's a donation link there. But we need we need funding for defending, what is essentially a fundamental part of the Internet's architecture. Yeah.

It's really interesting. I mean, there's an aspect of this. There there's two things that come to mind here. Number one is that, DNS providers don't have the same kind of platform protections that a lot of, let's say, social media contents or search engines do when, you know, we're not the provider of this content. The user is the provider of this content.

We're not responsible for it. And it seems like that kind of general hosting provider slash, you know, search engine slash social media protection isn't there for DNS providers. Well, I mean, it's I I can't make a statement on whether, social media is a content provider or not. That's that actually is a gray area that it's that's too difficult for me to read. I can, however I'm very I'm very certain that DNS providers do not host content.

Right? We we don't even know who the site owners are. There is no relationship. There's not even we don't even know who our users are or who the authoritative operators are. All we do is map names to addresses.

Right? Yep. Yep. We're very clearly on the side of that that is a mere conduit as the as the phrase goes. Right?

We are we are just like a phone operator. We are just like a a a network operator where those that traffic may travel through us, but we don't actually know anything about the either of the two destinations. But we do know that we're providing a good service. We're improving privacy and security. And we know that attacks like this where people are trying to block certain sites reduces people's trust in our service.

And so that is why we're trying to get away from that. We're trying to have these governments create rules that exempt recursive resolvers from these laws because, otherwise, otherwise, it will be a bad outcome for everybody. Yeah. Yeah. The second aspect of it, which I think is, like, really almost impossible, you know, you mentioned that, hey.

What about French citizens overseas and territories and jurisdictions and whatnot? I mean, I I face this problem with GDPR. I'm a European citizen living in The US. Technically, my understanding is that I should have GDPR protections applied to me. But if I ever try to bring that up with a US company that is, for instance, collecting my data that is not present in Europe, a, they don't know what the heck I'm talking about, and b, they'll fight me on whether I'm actually subject to those protections or not.

So I totally get the challenge there. I I I don't know, like, if you have any feedback to that, but no comment. I'm just saying it reminded me of that same situation. It's it's a they're interesting waters to be traveling in. And by interesting, I mean, really expensive.

Yeah. And, we need a clear answer, at the EU level, certainly. That's that's the area we're most interested in is getting the EU as an as an entity to say, okay. And they actually did interesting that the DSA, the Digital Services Act, actually has an a call out in there that says that recursive resolver operators are mere conduits. However, that needs to be then translated into a rule that can be applied in each country.

So we're, you know, we're we're in this gap period where that clarity isn't there. And then as the EU goes, so sort of goes the rest of the world as far as privacy. So we're hoping that that I wish that were true. GDPR still hasn't really taken root here. Yeah.

It's not always. But but there is a there's certainly a tendency for that. And so we're hoping to get that clarity soon. And so that, again, we can have recursive DNS left alone. Like, that like it's like telephone numbers.

You shouldn't constrains people from dialing certain telephone numbers. Like, that that that don't touch that. Yeah. Otherwise, you can do significant really, the again, the damage is trust. If you if you tell people, like, you're using the DNS, but, really, it's not the whole DNS.

You're creating a splinter net. You're saying, okay. You can reach all these websites except for these, which are prohibited. So now you've got a bifurcated view of the Internet. If you wanna go to these sites, you have to find some other way to do it.

And that creates, again, like, unintended consequences for these people trying to, to create censorship. Like, there there are actually big negative outcomes of that which, are gonna be a a result. So, anyway, this is a much longer conversation. I could go on for this for hours. But I know that we only have a limited amount of time and and, but it's it's a fascinating place to be and, quite busy.

Yeah. We've definitely gone over time, but I really appreciate it. And I'm sure our audience will as well. This is, again, a topic that I know, like I said, near, maybe not so dear just because of the frustration some of us have had in the past, but very near to the heart of of pretty much most of our audience and certainly myself included and certainly super critical for the operation of the Internet that we all use every day for every service that we use. I I I tell people, you know, APIs are everywhere.

You don't realize it, but, you know, everything that you're doing online is x number of API calls. The same is true, maybe even more so of DNS. It is just one of those critical underlying things that is there. So, John Todd, I can't thank you enough for taking the time to come on and join us today. For people who wanna learn more, we'll have the DNS iQOO, as I mentioned, just for a bit of fun.

But more importantly, we'll have Quad9 linked from the show notes. And for anybody who wants to learn more about some of these cases, is is the Quad9 site really the best place for them to go look? It is. It is. We've got pretty current updates there.

Awesome. Awesome. You can see the news section. You can see the about section. You can see the calls for donations.

You can see their corporate partners. Very open and transparent about everything that they're doing over there. John Todd, thank you so much for taking the time to join us on Modern Cyber today. Thank you for having me. Alright.

Bye bye. Bye.