Interview Transcript

Disclaimer: This interview is for informational purposes only and should not be relied upon as a basis for investment decisions. In Practise is an independent publisher and all opinions expressed by guests are solely their own opinions and do not reflect the opinion of In Practise.

I'd like to revisit a topic we discussed earlier in the year. At that time, Cogent hadn't yet completed the acquisition of Sprint. They finalized that in May. Now, six months later, I'm curious if you've had any interaction with them regarding the network and what your impressions have been, if any.

Yes, we have been interacting with Cogent and the impression has been positive. We are discussing specific routes in the Southeast and Mid Atlantic. These are the Sprint Wireline assets. These routes are valuable to us because they offer diversity. There are two use cases. There are some new sites that we're looking to connect, and some existing sites where Cogent could offer us diversity from our existing routes today.

You're referring to physical path diversity; same city pair but different path?

Correct.

What size of capacity do you need along these routes?

Several hundred gigabyte circuits.

I see. How did this come about? Did you have a specific need in the Southeast and Mid Atlantic, or was it presented to you?

Yes. The selling process is quite different for hyperscalers. It involves us approaching the suppliers and specifying our needs, such as a city pair or a facility pair. We ask them about their ANZ capabilities and what they can provide, whether it's a Wave request or dark fiber.

We interact with the Cogent account team to learn more about the assets. Usually, the suppliers will share a KMZ or the most up-to-date network map so we can see what they have. However, unless there's a specific project requiring certain connectivity, we don't make purchases. Our purchases are very much project-based.

We strive to be strategic and smarter, planning ahead when possible. For instance, if we know we'll need connectivity from Charlotte to Atlanta in six months, we'll start discussions now. This planning aspect, which we refer to as Regional Master Planning, is challenging and often reactive.

There are medium and long-term projects where we have more time, but with the introduction of AI and its use cases, our approach has become more reactive than we would like.

Are you conducting the zero 0DTR? I can't recall the exact acronym. It's something like signal loss along a route. Do you gather that data? How do you evaluate it? What's your workflow to complete that evaluation, especially if it's currently in progress?

We completely rely on Cogent for reporting aspects such as signal loss or latency. Unfortunately, there's nothing we can do directly, even though we wish we could. If we had field personnel, this would be a more human-intensive part where Cogent is untested and unproven for this type of service.

I believe we will end up entering an arrangement where we test certain circuits or routes. This will likely be a paid exercise, despite our preference for a no-cost solution. The time and effort involved on both sides, ours and Cogent's, to establish new wave circuits is significant. We would have to commit and take the risk.

This probably wouldn't be a situation where we don't have existing services through anyone. Instead, we might use Cogent to buttress and add physically diverse paths and test them. If it performs as expected, it may roll into our typical three or five-year lease. If we encounter issues, there would be a mechanism to continue trying to resolve them without paying in the meantime. If it's not working, we would terminate the arrangement.

That makes sense. It's a unique situation because it's a new service provider on assets that haven't been marketed before. So, what brought you to Cogent for these routes? Is it their physical uniqueness and lack of alternatives for your needs? Or is it better pricing, faster provisioning times? What made you want to go through this exercise with them?

I would say all of the above. There is some uniqueness to some of these routes. Even if we have existing connectivity today through Lumen, Zayo, and a few other regional players, depending on the location. We've had some pricing discussions, and pricing will not be a barrier.

The other significant aspect in this business is the delivery and circuit turn-up. Once we finish with the sourcing acquisition, contracting, and technical assessment, it's all about delivery. That's where they're making big promises, whether it's nine days, two weeks, or even 30 days. That's unheard of in our industry.

One of our questions, which we've started discussing, is how Cogent is going to support this from an organizational perspective. How is Cogent building out their field maintenance support? It doesn't have to be full-time employees; it can be a network of contractors in local areas.

Everything could go smoothly with the installation and provisioning, but the real test is when there's an outage. Companies like Lumen and Zayo know what they're doing. They resolve the outage or the faults. We have very strict Service Level Agreements (SLAs) tied to that, with time requirements as well as credits and penalties. The most important thing is to fix it and get it working. That's where we need more data from Cogent.

Indeed, everyone has Service Level Agreements (SLAs), and while you can get compensated for a service disruption, the credits don't actually resolve the issue, do they?

That's correct.

From my discussions with Dave on this topic, I understand that they are a significant buyer of dark fiber as well. They operate their entire IP transit network on long-term dark fiber leases, so they are well aware of how Zayo and Lumen perform in relation to their SLAs.

Dave has indicated that they are quite comfortable knowing that their performance will be on par with, if not better than, what Lumen and Zayo deliver. However, the specifics of how they will achieve this, whether through Full-Time Employees (FTEs) or third-party contractors, remains to be seen.

Having a better handle on inventory management, something that Zayo has notoriously struggled with, will likely help them identify issues more quickly than their competitors. But we'll need to see some real business ramp up first to confirm this, right?

Yes, understanding their own network and being able to respond quickly during the project's assessment phase is crucial. The Cogent team has been able to provide us with detailed information about their network quite rapidly. This is the room number. This is our suite in this facility. Yes, we can get there, or you can meet us, but they can take weeks on the audit and inventory part when we just want to identify if they have capacity.

I completely understand, especially considering the importance of reliability, consistency, and resilience of the network for hyperscale operations. I've heard from others that signing Master Services Agreements with hyperscalers can be quite a complex process.

That's a true statement. The process can be cumbersome due to the extensive agreements. We try to account for nearly every scenario we can think of, both operationally and legally. It's a lengthy process, but it's the same for everyone. Even with our current portfolio providers, we have to refresh these agreements every few years. The process can take even longer when we're doing refreshes where the contract vehicles are already in place, but we're just updating because it's been five, six, seven years. However, things are moving along with Cogent. The business we're doing with them now is different than what we've done in the past, particularly on the DIA IP side of things. But it's not causing any delays at the moment.

I understand that they're deploying the Ciena RLS across this network. Once they do that, they will be able to unlock faster provisioning times, higher speeds, and higher capacity, single wavelength routes. Knowing this, would it be better for you to wait for them to complete a lot of that work? Or does it not matter to you as long as you have the SLA? I'm curious about how customers respond in the marketplace when a company knows what the network roadmap and upgrade roadmap look like.

It's beneficial to know about future plans and upgrades. However, we're not going to wait for anything. If we can determine that the routes or circuits function correctly now, and we can manage any migration, upgrade, or maintenance, we won't wait. We have our own timelines, especially with the near-term projects that keep growing due to how busy we are. We don't wait for anything because things can happen. The upgrade, maintenance, or migration, whatever's going on, is always happening across the board with portfolio suppliers. We just have to keep going and manage through it. That's where vendor diversity and route diversity become really key.

What do you think your aggregate capacity growth will be? If you think about it on a compound annual growth basis over the next three to five years, what would you estimate your network capacity growth needs will be?

In terms of the network, it mirrors our data center capacity and growth. What we're really supporting is our overall data center capacity gigawatt expansion in specific regions and sites. On a percentage basis, it's a robust 20% to 30%. We're nearing capacity in several metropolitan regions where, due to our design, we need to add extra network gateways solely for the network. This is separate from our data center capacity.

We're transitioning regions from the 528-megawatt size to gigawatt regions. The fiber or connectivity must expand to meet those needs. We're discussing waves and dark fiber routes where we're adding double-digit strands to existing routes on the dark fiber side. These are multiple long-haul routes across the country. We refer to these as our ladder routes along with our east-west routes.

On the wave side, at any given moment, we're purchasing multiple 100GB waves. We're beginning to purchase 400GB waves, which are easier to manage on a circuit-by-circuit basis. When setting up new sites, we require several terabits of capacity from day one. Depending on the schedule and timeline, wave service is often our only option in certain locations.

I'm curious about Cogent's pitch. They plan to connect all their 800 carrier-neutral data centers, which, in my mind, is like offering direct flights everywhere, rather than going to a hub and then using a Metro Wave or Metro Dark fiber from the hub to the local data center or the other carrier neutrals in that area, that's an oversimplification. But that's how I've understood it and I'm wondering if that matters to you. You have a complex network architecture with metro dark fiber, metro waves, long haul dark fiber, long haul waves, and everything in between. Does the value of direct facility-to-facility connectivity matter to you, even though you currently have to piece it together through multiple contracts?

Yes, that matters to us. It's not only more efficient, but technically it's a better way to build a network when we can go direct. As you said, the non-stop flight. It's not just about the non-stop aspect, we know the exact route that the flight is taking all the time. We know how long it will take all the time. So, latency is crucial. If we can reduce latency by not having to go through the carrier hotel and cross connect, thereby avoiding potential points of failure and additional complexity, that's definitely an advantage. Being able to go direct and land at a specific facility or site because Cogent is on net there now is beneficial.

Is that what they're building? Is that what they're enabling or are they not?

Our discussions primarily revolve around public data centers. These are located in Equinix buildings, or Digital Realty buildings. We might be in the same building or the next one over on the same campus. We're not at the level where we're discussing with Cogent about building into our data center or location, even if they're still offering us wave service. We'd be discussing dark fiber at that point. The data center hubs in Charlotte, Atlanta, and Seattle are the major carrier hotel locations where everyone else is.

The last mile from the carrier neutrals to the proprietary facilities would be handled differently in some cases?

Yes, in some cases. These are in the metro regions. We have large capacity facilities in certain areas of the country that aren't near these carrier hotels. That's where we have dark fiber routes that are specifically constructed and delivered to service those.

Which can be cumbersome.

Yes, it doesn't happen quickly. But it's the most efficient for us due to the bandwidth and capacity requirements. A wave service or system just won't work. Economically, we purchase our own Ciena gear, light up these systems, and operate them ourselves. Not only is the 26 terabits of capacity between these sites required from day one, but financially, the business case is easy to solve for when it's that much capacity and we light up these systems ourselves.

At some point, the owner's economics make more sense due to the infinite scalability.

That's right. There's no one else there. There are local players and we have these conversations. It's unique to the hyperscalers. You talk to others. It's likely a similar story if you're talking to enterprise customers that purchase Internet service and waves as part of their network requirements. Hyperscalers, for better or for worse, are in a league of their own.

I was listening to Lumen's earnings call the other day. They mentioned a balance sheet issue and are going through debt restructuring. Their stock is around a dollar, and the public nature of their debt restructuring conversations with creditors caused them to lose some business in the quarter, or maybe the year. I've had this general question about the existing duopoly in this transport market between Lumen and Zayo. They're financially very weak. Zayo has about eight turns of leverage, and I don't know how much longer Lumen can keep kicking the can down the road before they have to do a full restructuring. But when it comes to the operational parts of the business, how are they performing? Do you feel any differences from these companies when it comes to getting what you need from them or their willingness to work with you and sell to you? I'm curious if you feel any of that distress in the market as a customer.

Our internal discussions increasingly revolve around the weakness in Lumen's balance sheet and their overall management as a company. This is of concern because we heavily rely on both Lumen and Zayo for our core network infrastructure. It would be disastrous if they were to suddenly cease operations.

However, I don't believe they will ever shut down completely. The equity might get wiped out along with some of the debt, but that's about it.

A major restructuring or bankruptcy could occur, but operationally, it's unclear how that would affect things. In theory, it should be fine, but it's uncertain. This issue is becoming more prevalent. We have no other choice. Even before AI entered the scene 14 to 16 months ago, the network demand, data needs, and capacity requirements of our business were the primary drivers of our expansion and growth plans. This included both expanding current routes and building new ones. There was an expected softness from the enterprise vertical or the enterprise customer late last year and early this year, which was evident in Microsoft's reported numbers.

Are you referring to the Cloud optimization?

Yes, the concept of doing more with less. That's no longer a factor. The introduction of AI has only accelerated everything. However, we rely heavily on Lumen and Zayo for our backbone.

It makes perfect sense not to rely too heavily on a company in financial distress. If Cogent offers a viable third alternative as a national long-haul provider, do they stand a chance of capturing a significant portion of the business growth? It's been said that no one in the hyperscale ever turns off a circuit or switches providers due to the constant growth. If your technical testing of Cogent's infrastructure proves successful and it meets all your needs, do you think your market share allocations for long-haul would change?

We are considering that. Yes, there is potential. We'd welcome a third national option. The current issue is whether Cogent can meet our needs. We're primarily concerned about ourselves. Will they expand to areas where they are currently near-net or not close to some of our facilities? For instance, in Phoenix, where we have our own data center campuses and are continuing to build, we only have one choice. Cogent is around the Phoenix areas, but will they invest, say, 30, 40, 50 or even 100 million in that specific region to become an on-net provider for hyperscalers?

I'm not certain, but my guess would be no, due to the high customer concentration risk on a significant capex program. It's a big bet on a single customer. I believe their modus operandi is more about connecting to the highest traffic facilities, regardless of whether they are DIA, transit, or waves. If a customer needs a last-mile connection to a proprietary facility, I don't think they would be the ones to provide it.

That's correct.

Is the same true for Zayo and Lumen? Are they investing 50 million in capital to connect your proprietary data centers to the connectivity hubs? Perhaps that's why they're in financial distress.

They do get paid well. I can tell you that our business with them is quite healthy. To really win backbone, or long-haul backbone share, I believe that's what it would take for Cogent to capture a significant portion of the market from the other two.

That's not to say we're not involved in commercial leases or lease-to-own situations, especially in these metro areas. There's a lot we could do. However, Cogent would need to show some willingness. Perhaps it's not a large construction or build, but they would need to meet us if they're a mile or two or a few kilometers away in a certain metro.

For instance, in the Bay Area, San Jose, if they could come up to Santa Clara, get the permits, and do the last mile build or last couple of mile build, and place their equipment in our facility, that would be most beneficial to us. That's how they could capture market share. We're not talking about 30 or 40 million, maybe it's a one or two million build, something like that. And it takes about twelve months.

If there's already a single provider, could they theoretically buy dark fiber from that provider on the Metro route and then just do it through there and they give you the whole route, some of which is owned and operated, some of which could be leased?

They could, but what would be the point? As you move out of the Metro, it might become diverse, possibly from Zayo, for example. But you still have that single point of failure coming into the facility or campus. That's where there's going to be a fiber cut, and that's where we're going to have an outage, where there's a single point.

Let's discuss this AI burst that you're seeing. Could you give me a high-level overview of what it looks like and how AI is impacting your business?

To put it succinctly, it has accelerated everything. This is an understatement to say it has been a tailwind. It has expedited our build plans, expansion plans, and timelines. We have specific requirements from customers to deploy AI clusters in specific locations because that's where they need them. When dealing with learning and inferencing use cases, you don't want to transport large amounts of data across the network unnecessarily. You want it close to the compute, close to the customer clusters, and so on. This has been beneficial for hyperscalers and us. It has also accelerated the demand for waves. Even if we're talking about 90 to 120-day delivery times, that's excellent. We can get online much quicker than building out these dark fiber routes, which can take 12 to 15, or even 18 to 24 months in some cases, depending on the location. In general, it has been a significant business accelerator for us, our suppliers, and the industry.

That's interesting. It's not surprising, though. It seems like this is driven by customer demand rather than building ahead of anticipated demand. Is that correct?

It's a combination of both. The short-term projects are more customer-driven, while the medium and long-term projects are supply-side generated or initiated by us. We recognize that the capacity and compute needs in certain areas of the country will continue to grow. We need to reinforce our current network, our regional network, and build more sub-regions and facilities to continue adding capacity. This refers to data center capacity. We're talking about hundreds of megawatts across the country. Some of these plans extend to the end of this decade, into 2030. That's the time horizon we're planning for. We're not starting those builds now. My team is working on contracting and sourcing builds for 2025 and 2026.

But in terms of planning, we're looking at the next decade. The AI accelerator and AI use cases are at the top of the list for why we need this capacity. We're still trying to optimize and things like that. But that's one aspect of it. The other aspect is our cloud infrastructure, which continues to be taxed and needs to grow. As you mentioned, the resiliency, redundancy, and reliability always need to be enhanced. Our SLAs and latency commitments are becoming stricter, and the latency demands are increasing. So we need to be closer and closer to our end customers. This is why we build these edge deployments. But it's really about the compute and the storage. How close can we set up these massive clusters to our end customers?

This isn't my area of expertise, so I might ask some basic questions about how this works. For instance, if I'm Procter and Gamble and I'm trying to use AI in my business, using Azure and OpenAI or similar, what does that look like? What do you mean by setting up compute clusters closer to the end customers? If they're already in the Azure cloud and obviously have a mix of on-premises and cloud, could you explain what that network architecture looks like for AI applications?

Yes, their infrastructure is in the cloud, but the cloud is located in specific availability zones within a certain region. This is likely closer to their on-premise infrastructure or their core infrastructure. It's not located randomly. Depending on what they're running, they may be sensitive to latency. Whether it's customer data or marketing, given that they're a product and marketing focused company, there may be some AI use cases.

They would want to establish their AI clusters where their core compute and storage nodes already exist. The reason is, you wouldn't want to transport petabytes of data across the network. This not only incurs high costs but also ties up the network and slows things down. Despite the efficiency of the latest Nvidia H 100 chips, if the transport layer of the network has to move data over long distances, it introduces latency and lag in response or delivery time. Therefore, it's crucial to set up close to where the core operations are.

If they're already in the cloud in an availability zone or region close to their headquarters, is the infrastructure at the Azure/AWS/GCP level housed in different facilities, or is it within the same clusters of data centers and proprietary facilities?

If there's space and power available, it would be in the same facility. The issue is that there's no availability in our current facilities.

So there's no space and power. What happens when you get a burst from AI? How do you meet the power needs? How do you find a solution?

We are eagerly pursuing new projects, some of which are short-term, within the next 12 to 18 months. We can establish a new 32-megawatt facility in about nine months, but the fiber won't be available for 18 to 20 months. This is where wave service can be beneficial, as we can set up waves within nine months if the data center is ready.

However, there's limited capacity in existing facilities like those of Equinix, Digital Realty, Vantage, and STACK. These data center companies are also expanding, and we're competing with them to secure land and power. Power is a constraint everywhere. The utilities in these states and metros are not designed for hyperscalers; they primarily serve residents.

This has resulted in a long lead time engagement where power won't be available for several years. Or if it is, Equinix has already secured the next, let's say, 72 megawatts that will be available in the next twelve months. We're grappling with these issues to establish additional clusters to serve our customers.

It's a struggle in some of these densely populated metros. In theory, it seems simple to set up these clusters, but there's no space. An AI cluster, for example, requires double the power of a traditional server cluster. The cooling requirements are also much greater. Some HVAC systems can't handle that amount of heat, so we're considering liquid cooling in some locations.

We're now considering 8 to 10-megawatt facilities that we wouldn't have even thought about twelve months ago. Financially, it takes millions to establish a new facility and get the dark fiber in there. We used to consider 32 to 36 megawatts as the minimum size, but we're reducing that by two-thirds or more, just to capture any available capacity in these metros.

Coming back to Cogent, they acquired a number of data centers and technical facilities from Sprint, which had a lot of dead equipment, I believe over 20,000 racks. They're clearing them out. Dave mentioned that every data center operator wants them, likely for the reasons you've just explained. Even though they're not the latest and greatest, they offer about 160 megawatts of power across these facilities.

While it's not a lot, it might be enough to serve as a temporary solution for your current needs. Have you discussed this with them? Are these facilities potentially interesting to you, or are they too small, too dispersed, and too outdated?

Dave's better option might be to sell directly to the likes of Equinix and similar businesses. There would be significant retrofitting and work involved, which is something we haven't discussed with Cogent. If there was a facility in a city where we desperately needed capacity, we might consider it. We would treat it as a new site, a tear-down site, or a site with existing infrastructure that we could retrofit.

However, Dave entering the data center business might be too much of a stretch. He could certainly command a premium, especially if these facilities, even if not currently operating, have secured power and no issues with energy requirements. It would likely involve a retrofit and possibly a fiber overhaul. This could be a good business line to raise some cash, at least that's how I see it for Cogent.

The intriguing aspect of this acquisition is that they have just acquired these fallow assets. There are data centers with equipment that's around 20 years old, but the racks are still powered on. The fiber network, which was vacated by core Sprint and T-Mobile traffic a few years ago, is essentially empty. The majority of the Sprint contracts are off-net. They're not even using it. I think it's about 7% on-net circuits for the Sprint business.

They had two different things, a proprietary asset base that was used internally and then no longer used over time, and they had a B2B enterprise business largely built outside of the Sprint network, which is one of the most idiotic things I've ever heard. It’s Cogent's opportunity to combine network assets and put more of the business on net, which will save money. Then they will see how to use the fiber network, which is essentially empty.

They acquired a couple of thousand buildings, but only around 50 of them are suitable for data center or colocation applications. It's a fascinating use case. I didn't realize that the power consumption is at least twice that of a standard x86 cluster or standard cloud computing. The lead times on these things are usually many years. I'm sure you're used to thinking many years in advance compared to what you're dealing with in AI right now.

How do you handle this? Do you think you can fulfill it? Are you concerned that the capacity, power, or fiber might not be there? How do you solve these problems? Is it a matter of money? Could you spend enough to solve them if necessary, and would the returns be there? How do you think about that? How do you manage that?

It's less about the financial aspect, although that is still part of the business case. We've canceled projects because the investment wasn't justified. However, the real issue is the speed at which we can deliver the required capacity. As I mentioned, we're seeking smaller sites that we haven't previously considered. It's a scramble to find any available capacity, particularly in metropolitan areas, and to make it operational as quickly as possible. That's our short-term solution.

In the medium and long term, we've increased our efforts in land acquisition. A team separate from mine is tasked with finding new usable land. We're not only competing with our peers but also with data center companies who are trying to do the same thing. We're also working directly with utilities to determine which substations need to be built and what transmission and distribution infrastructure is required.

It sounds like a whole modern infrastructure layer. The data center footprint was built in traditional hubs for core networks like Virginia, the Bay Area, and Seattle.

That's correct.

How do you envision the future? How will the cloud footprint compare to the telecom footprint?

There are a few concentrated areas. For instance, in Northern Virginia, we're exploring Southern Virginia, although we're still very active in the north. However, new capacity is limited. We're looking at Loudoun County and other counties to the south. The fiber infrastructure then becomes a factor. There’s nothing immediately available in Seattle. The Bay Area has some capacity available, but we're being very selective.

I know there are some data centers in Eastern Oregon along the river. I'm not sure if they belong to Microsoft, AWS, or Google, but there's a cluster of facilities there. Considering that access to power is often a limiting factor, along with space and telecom connectivity, do you foresee cloud and AI facilities being located more in suburban or rural areas over time? Or will they still be centered around major metros?

It's going to be a mix of both, but the core infrastructure will likely be located in more rural or suburban areas because that's where it can be supported. Eastern Oregon, for example, is a popular location.

To wrap up on Cogent, how long does the evaluation process take for you? If you have routes that you want to acquire from them, what does it take to go from where you are today to awarding them a contract, which then takes them time to provision?

I'm uncertain at this point. I believe it will take more time. I'm hopeful that we can conclude matters by the first quarter of next year. However, it largely depends on the specific project.

By the first quarter, you would have completed the technical evaluation, possibly conducted some tests, and decided to proceed?

Yes, that seems likely.

If the first project is successful and approved, I assume the process to gather all the technical details for each subsequent route would be quicker. You could work with them more efficiently, correct?

In theory, that should be the case.

Excellent. It's always enlightening talking to you. I always learn so much from our conversations. There's so much happening in your world, it's an exciting time. My interest in Cogent is evident, but the entire cloud infrastructure space is fascinating. The technical support you provide is far from trivial.

Thank you.

There's a lot of components you need to assemble to keep the modern world functioning. It's a marvel of technology and infrastructure.