In the world of political and media misinformation that is the NBN, an important issue, that hasn’t been fully addressed, is “How fit for purpose is Australia’s copper network?” This seemingly mundane and tedious question directly affects tens of billions of dollars in government spending. How?
The bulk of the Coalition’s NBN alternative policy uses the existing copper network to get the internet to your home or office – a system called Fibre-to-the-Node (FTTN). It aims to do this in order to follow the broadband upgrade path in other countries – where incumbent Telcos (foreign equivalents of Telstra) are upgrading their copper networks to ‘sweat the last ounce of value’ from them before the inevitable switch to Fibre-to-the-Home (the system that Labor has jumped straight to).
But there are unique challenges with following a FTTN policy in Australia – not least of which is that neither the government nor NBN Co owns a copper network and so would have to somehow acquire Telstra’s – we’ll deal with that in a future article. For now we’re focusing on the condition of the main copper network.
Why might it be a problem?
The potential internet speeds offered by FTTN are undisputed. However, in order to realise them, the copper network needs to be in good condition (in addition to homes being close to exchanges and having the right wiring). Deutsche Telekom, for instance, goes to great (and expensive) lengths to ensure its copper network is running optimally and has subsequently become a poster child for FTTN’s potential. Meanwhile, countries like the UK only offer FTTN in areas where the network can support it (less than half of households) and promise minimum speeds of only 15Mb/s.
The Coalition is promising a minimum 25Mb/s download speeds to all Australian households by 2016 with this rising to 50Mb/s by 2019.
But these promises are threatened by considerable evidence which questions whether significant portions of Australia’s copper network are fit for purpose with regards to a full-scale FTTN rollout. The evidence ranges from voluminous customer anecdotes and historical media coverage, through observations by Telstra engineers, Unions, telecoms experts, cross-party parliamentary inquiries and statements made by Telstra itself, all the way to photographs of appalling maintenance jobs.
There are definitely problems, but to what extent? This is important to know because in most instances, we won’t know the potential speed offered by FTTN until the network is turned on. There can be potential for remediation, but that could mean potentially-colossal cost blowouts. The Coalition has budgeted for some remediation, but nobody knows for sure how much is needed. In an industry where Telstra and Optus collectively spent some $50bn on their networks over the past decade and Telstra’s annual copper network maintenance costs are commonly regarded to be range from several hundred million dollars to over a $1bn, the assumption that everything is fine needs to be tested.
What follows is the best evidence available in the public domain.
In Telstra’s own words
Telstra currently remains tight lipped about the state of its copper.
But that wasn’t always the case. The first significant quote comes from the April 1998 edition of Australian Communications. At the time, Telstra wanted to build a Fibre-to-the-Node network as a competition blocker and get rid of ADSL-based internet. Telstra’s Group Manager of Corporate Affairs, Martin Ratia, said: “We are not going to keep archaic technology going just for a couple of service providers who do not want to upgrade… We cannot keep a copper network in this country for half a dozen ISPs who want to make a quid.” Around that time, Telstra CEO, Frank Blount, forecast that the customer network would be all Fibre by 2010.
In 2004 a spokesperson said, “Telstra is not interested in pursuing VDSL. We are not trialling VDSL. We see fibre to the premises as the most likely technology to support very high speed access services of the future.”
These are telling views of the company’s regard of its network but don’t explicitly describe its condition. However, the company spelled things out, ten years’ ago, in 2003. As Alan Kohler wrote:-
A month ago, before a Senate committee inquiry into broadband competition, Telstra’s Bill Scales and Tony Warren rather let the cat out of the bag.
Warren, group manager, regulatory strategy, told the committee: “I think it is right to suggest that ADSL is an interim technology. It is probably the last sweating, if you like, of the old copper network assets. In copper years, if you like, we are at a sort of transition – we are at five minutes to midnight.”
A few minutes later his boss, Bill Scales, attempted to bury this bit of candour: “The only point of clarification, just so that there is no misunderstanding, is that when we think about the copper network, we are still thinking about 10 years out. So five minutes to midnight in this context…”Dr Warren (chiming in): “Doesn’t mean five years.”
Mr Scales: “It does not. It could be 10 or even 15 years, just to get some context into that.”
More recently, Malcolm Turnbull published an FAQ which directly asked, “Isn’t Telstra’s copper network too degraded to deliver speeds of 25Mb/s to 100Mb/s?” Current Telstra CEO, David Thodey, is quoted in the answer but his observations deal with FTTN speeds in European markets.
Indeed, Telstra has a habit of sidestepping questions of this nature. The company also recently suggested that its copper network is fine because Telstra spent many millions each year maintaining it and that faults are around just one per cent although that figure deals with reported telephone faults and bears little relevance to the issue at hand – more on that here and here.
More recently (since the Coalition committed to procuring access to the copper network) Thodey said, “The copper has been going well for a hundred years. I think it will keep going for another hundred.” No further evidence or explanation was given for this statement, but we’ll revisit it later.
Having to guarantee broadband services seems to worry Telstra. Just late last year it submitted to the ACCC that it refused to offer a Naked ADSL service (internet without phone line) on account of (as ITNews puts it) “increased fault rates” from “corrosion” caused by “moist air” and “actual water ingress” to the cables which make up its infrastructure,” making the practice unworkable.
What the papers say
In 2008 Joe Hildebrand wrote a fascinating article in the Daily Telegraph which slammed the state of Telstra’s copper network:
Thousands of people are having their phones cut off every time it rains because cost-cutting by Telstra means the lines are no longer waterproof and sometimes protected only by children’s lunch bags…Workers say Telstra’s squeeze on technician numbers over the past 18 months – a cut of 2600 nationwide, including 550 in Sydney – has meant up to 50 lines in each underground footpath box are not properly protected.
In many cases technicians, overloaded with the 500 per cent increase in workload, are forced to do hasty patch-up jobs with tape and plastic bags. They then put in a request for an outside contractor to complete the job but often this is not done in time, leaving the lines exposed to water…
“These rising high volumes of faults are caused by Telstra’s ongoing program of retrenching skilled communications technicians and major cutbacks to the maintenance of Telstra’s copper cable network,” [Communications Electrical and Plumbing Union – CEPU] assistant secretary Steve Dodd said.
“Hundreds of skilled communication technicians have been made redundant in Sydney over the past 18 months following Telstra’s CEO Sol Trujillo’s announcement in 2005 to reduce its workforce by 12,000.”
At a more-local level, The Coffs Harbour Advocate tells of a carer and his seriously ill wife who, “often has to call the ambulance and never knows when he may need to make an urgent call.” But the line frequently stops working and is fixed using “an inverted blue plastic bucket placed on top of the junction bundle; Glad and home brand plastic bags of varying sizes, from small no frills lunch bags to ‘top-end’ Glad bags, press seal variety.”
The Telstra area manager “Defended the plastic bags, saying it was normal practice to use them to keep water out.”
The plastic bag fix is such that technicians routinely refer to the Telstra network as Baghdad. Tim Ashford, a highly-qualified, Telstra-accredited engineer who recently retired having started in the industry in 1961, tells a tale that has come to sound very familiar over recent years, “Contractors go in there and get paid by the job, so they just want to be in and out. It takes time to do records and to seal the joint properly so they don’t do it.”
CEPU itself recently released this extraordinary directive to members:
No Contractor can guarantee his work for 14 days anywhere across Telstra’s plastic bag “BAGHDAD” network.
Telstra cannot guarantee the performance of their network to their customers, yet they have an expectation contractors should guarantee their work reliant on that network for 14 days.
The situation is totally unacceptable on ISGM Contractors and the CEPU advises members that NO contractor should participate in the handing out of the 14 Day Customer Card.
The release also calls Telstra’s network, “A network that is now in a state of disrepair.”
What the Union says
When I asked CEPU about the ‘Baghdad’ issue, the floodgates opened. Assistant Secretary of the NSW Postal & Telecommunications Branch, Shane Murphy instantly labelled the copper network, ‘a disaster’ and gave a lengthy explanation as to why. He said if I saw the state of the pits that Telstra own I’d find it ‘amazing” and I should “see the text messages and photographs” that his members keep sending in.
“Unfortunately Telstra’s industrial relations and ideologies, a number of years ago created the network that it’s [got]. It’s not through any shape, form or the workmanship of our members. These guys were driven by Telstra on performance management under the Sol Trujillo regimen during the Work Choices era and they were absolutely driven hard to get as many jobs per day done at the expense of quality. It was all about quantity. They were pushed and pushed and their performance targets and measures were continually pushed up…
In the meantime they were placed under pressure to achieve or they were then called into meetings. At the same time Telstra got rid of its contract and internal inspectors who would perform routine inspections across the network to make sure the work was being done in accordance to policy and procedures and they went at the same time that they rolled out this policy of, ‘Get to your next job, get to your next job, get to your next job,’ at the same time as cutting a number of jobs as well.
“At the same time they implemented a wage under Work Choices. They paid people under a job point system instead of hourly rates which we had under the enterprise agreements. That was pretty much a ‘pay per job.’ They introduced independent contractors into the system. These guys were paid per job, some years ago at a reasonable rate, and over the last few years those rates have diminished down… It’s been slash and cut, the network, and there’s been no maintenance done for some years. And [the end] result unfortunately is the state of the Telstra network that you have right now. “We don’t accept the Coalition’s alternative for the National broadband Network… The part of the network that they suggest they want as their part of the policy… most of the problems for the customers across Australia… it is from around where the pillar is – or where the new node will be – through all the copper network that we’re talking about being ‘Baghdad.’ But the expectations of the speeds on the internet would likely be unchanged because they’re going through the same [part of] the network that I’m talking about.”
Since this statement, CEPU have further expanded on their claims. You can watch them demonstrate thecondition of the network on Lateline here.
While the contempt for Telstra is dripping off CEPU and it’s simple to point at partisan politics, this needs to be tempered with the fact that CEPU members are the ones who are hands on with the copper and they have a treasure trove of evidence supporting their claims.
Outside Plant Records
Telstra’s optimistic reliance upon contractors to perform best practice repairs has caused other issues that directly affect a FTTN rollout. In an ideal world a Node cabinet would replace an existing Telstra turret and you connect the old wires to the new cabinet. However, this relies on the copper wires going where they are supposed to.
A senior technician who is currently rolling out the NBN but performed an audit of Telstra’s copper network some years ago – when Telstra itself was considering a FTTN rollout – told us that over 30 per cent of the records were wrong. He said that the only way to tell which copper line went from the turret to each house was to have teams of engineers (one at the turret and one at each address) test each line individually to check that it went to the right place.
He said this made the process of performing a “one-to-one translation” of lines from the current pillars to FTTN cabinets, completely impractical; pointing out that Telstra has some 65,000 pillars around Australia with many handling communications for around 200 premises.
Because of faulty pairs, ageing, water intrusion, changes and split-redeveloped blocks, redundancy in the cables is now much less than the original ‘near 100 per cent.’ It is now closer to zero.” Now add to this the adds, moves, changes, and the repairs performed by contractors who have no ownership of the infrastructure and who have not updated records for over five years. The result is we have no redundancy and a 30-plus per cent error rate in the records for the last kilometre – the pillar to the home.”
His summation of the network was that it was, “Rooted.”
On this evidence, confirming exact numbers of faulty records appears all but impossible and it’s unlikely that Telstra could accurately know itself without a new audit. It begs the question, should public money be spent procuring access to the copper network without an audit being done first?
Furthermore, assuming that significant remediation was required, under what conditions would contractors be expected to work and who would pay for them?
We’ve shown a few here, but there are significant collections which can be found on Delimiter and CAN Of Worms. Delimiter makes no bones about its pictures representing the worst of the worst and has even made an attempt to show some positive examples.
Image source: CAN of Worms
There are numerous pictures like this, from around Australia, online. The big question is, just how representative are they?
The Gel debacle
Telstra used waterproof gel to seal its copper network joints and keep out the water. This caused several issues, all of which are poorly documented. Firstly a batch of the gel immediately corroded around three per cent of the joints. Telstra acknowledged and dealt with the matter. The problem was that the gel also sealed existing water into the copper cables leading to additional corrosion. We’ve heard a great deal of anecdotal evidence from various Telstra Technicians regarding this, but few have matched the following erudite, all-encompassing description from one, recently-retired Senior Technical Officer who wishes to remain anonymous:
Everyone in management is denying there is any problem here, but it’s a huge one.
“Seal the CAN!” That was the catch cry by managers about 15-20 years ago, The CAN is [the copper] Customer Access Network. We had managers stand up and gleefully tell us that this would mean the end of us all, as soon as everything was sealed they would get rid of everyone. They were so sure that they put in place systems to track everyone’s use of the product and were following us around to make sure that it was used. This was from top management all the way down. If lower-level managers weren’t policing it they were in the gun. Three strikes and you’re out was another familiar catch cry, if they found you failed to use it on three joints you would be sacked on the spot. That was the policy and so the gel was poured into hundreds of thousands of joints all over the country.What is it?
We were told it was just a vegetable oil to which a catalyst was added to make it solidify and gel, the raw product certainly smelled like vegetable oil, there were some safety concerns about it though and special handling protocols were set up… The cured gel could be crumbled off a joint that needed to be reopened fairly easily but it was a messy job… and just the act of crumbling it out from around wires and joiners often lead to wires breaking and faults being put on. So you would access a joint, crumble the gel from the joint, test or repair or whatever was required then pour more gel back in, then the next day someone would need to access the joint again to fix the fault you had unknowingly put on.
What does it do to cables? Because of the heat and humidity the problems first started to appear in
North Queensland and the NT. Workers started to report that this stuff was actually ruining cables, these reports were ignored and if anything we were subjected to even more extreme measures to make sure we used it.
A couple of years later we started to notice that it was happening in Victoria as well, management continued to ignore it and escalated surveillance measures again because they thought it was a plot to keep our jobs. It took probably 10 or more years before it was accepted that it was causing problems, then several more years before we were allowed to stop using it. We were never officially told that it was a problem and to stop using it, managers suddenly were just very quiet about it.
To give you an idea of what it actually does to a cable I first need to describe a couple of typical cables. First is a modern grease filled cable. When the gel is poured into a joint with this type of cable the gel fills the joint and unless water gets into the joint everything is OK. Unfortunately there are lots of older cables that aren’t grease filled, when a joint or cable goes faulty, water will fill up the cable. Apart from causing some minor changes to the electrical characteristics and crosstalk it will usually have very little effect as the wires are plastic insulated. Now we pour some gel into a joint with a mixture of filled and unfilled cables (most joints).
In the unfilled cables, the liquid gel will often travel down the cable and fill up to three metres of cable before solidifying. Where the gel and water meet a reaction takes place I presume from the action of bacteria because the resultant mess smells disgusting, an acid or some by-product attacks the plastic insulation of the wires completely destroying it, then electrolysis corrodes all the wires. As you can imagine because this is happening up to 3 metres away from the joint it becomes very difficult to fix, often needing new lengths of cable, new pits and more joints in the cable.
When you are driving around the countryside keep your eyes open for pits with guards around them and a joint taped to the guard, often wrapped in a plastic bag – Gel [was the cause]. Often you will see two large white PVC pipe joint covers within a few meters of each other, two joints in close proximity because a new short bit of cable had to be spliced in – Gel.
This is not a minor problem despite what management say, there are hundreds of thousands of joints filled with this stuff all over the place, any of them can go faulty any time.
A similar tale was relayed to the Senate via CEPU. More descriptions can be found here.
What the government says
Nine years ago, the large, cross-party, Environment, Communications, Information Technology and the Arts References Committee repeatedly slammed Telstra’s upkeep of the copper network:-
The principal weaknesses relate to… inadequate repair and maintenance by Telstra of the network; and declining investment…
There is substantial evidence that Telstra’s fixed line network is deteriorating due to reductions in staffing and inadequate capital expenditure…
The Australian Communications Authority has been a reactive regulator. It has not been in a position to be proactive in important areas, such as monitoring the state of the Telstra network and requiring that infrastructure be upgraded where necessary… The Committee does not believe that Telstra has given sufficient weight to its role as the guardian of the CAN in the past…
Among the recommendations, which included a move to Fibre-to-the-Home, it said:
The Government should require the Australian Communications Authority (ACA) to conduct an independent inquiry into the state of repair of Telstra’s customer access network and the Government should, if necessary, use its powers to direct Telstra to bring the network up to an acceptable operational standard. As a part of the inquiry the ACA should examine technical standards and regulations, including those relating to preventing the ingress of water into CAN cables, and amend those standards and regulations so as to protect the physical integrity and ensure adequate maintenance of the customer access network… The Committee does not believe that Telstra has given sufficient weight to its role as the guardian of the CAN in the past – on which all other access is essentially dependent.
It does not appear that the committee’s recommendations were acted upon.
What the FTTN expert says
Dr Paul Brooks, is ex-CTO of TransACT and deployed Australia’s only, large-scale FTTN network.
In terms of line faults, most of the fault issues are out in the streets in the runs from pillar to house where open joints, cracked joint covers, wires hanging loose from poles, water in pits, wires twisted together (rather than being soldered etc) all contribute to poor ADSL signals, and sometimes even poor telephone audio – and this is precisely the section of lines that we are retaining and hoping to use VDSL2 over. For many people, it will work fine – for some, it just won’t.
The ADSL family of technologies (including VDSL2) is pretty robust at working around a reasonable amount of degradation, including bridge-taps, the odd badly connected joint etc. Sure speeds drop (and distance also) as those affected frequencies are blocked out, but other channels get around the impairment, so you can usually get some form of connectivity, even if you can’t predict the actual synch speed beforehand. A real faulty line though may completely fail to carry DSL signals, even when a baseband audio analogue telephone service works ok – and these faults are usually in the sections we will be retaining and re-using for VDSL2.
Tim Ashford plays every angle. While he agrees that some large areas of the network are “Rooted” he says that the copper lines, in many places, are actually in good condition (could this be what Thodey was referring to?) However, he goes to great pains to say that it’s the joints that are the main problem.
At the same time he is on something of a crusade to get Telstra to adjust its fault testing practices which are too ham fisted to pick up the small faults which cause service problems. The crux is that Telstra keeps charging customers $130 to check line faults which can’t be picked up by existing practices. He’s literally spent years campaigning Telstra to adopt a more time-consuming fault-finding method but to little avail.
His great fear is that if Telstra ends up managing the copper network for NBN Co, that similar lackadaisical testing procedures would occur. His focus on the fault finding issue makes it difficult to assess the state of the network as a whole: parts are unusable while the rest would work with proper remediation is probably a fair summary.
It is John Lindsay, CTO of iiNet, one of Australia’s largest ISPs, that commands the counterpoint. He points out that David Thodey seemed pretty confident in recent media reports and says that he’s seen Testra’s cables in ducts as part of rolling out fibre for various projects.
“The cable downstream of the pillars is often relatively modern. It’s mostly PVC sheathed and insulated and usually filled with an oily jelly that prevents water moving through the cable if the sheath is physically damaged.
“There are a series of joints along the cable. These are a weak spot in the system but if the joint housings are correctly used and maintained they are generally reliable. Yes, there are some great photo examples of where this has gone wrong but equally there are examples of joints in fibre networks that haven’t been looked after properly. “The cables from the pillar to houses are capable of being maintained in good order and much of that part of the network is visibly in good order. Underground PVC insulated copper cable should last centuries.”
He is at pains to not get political but points out that, “Fibre-to-the-Node is rational to be deployed in some areas and is easily capable of being kept in service for another few decades.”
His view is not shared by his boss, however. Michael Malone, CEO of iiNet, recently told a room for of telecoms journalists that it was indeed the last-mile copper that was in the worst state and focused on the point that it was therefore unsuitable for applications like Telehealth owing to reliability issues. He subsequently told ABC Tech:
“In my opinion, it’s a question of reliability. The copper network was built for voice and now has an average age of about 35 years. It can be affected by everything from corrosion to the weather. This can lead to true line faults, where the phone service just stops working, or more commonly to “dropouts”, where the line will spontaneously disconnect for a 10-20 seconds, then reconnect again. About 10 per cent of our customer base experiences dozens of dropouts per day.
When the line does have a complete fault, fault finding can be very difficult. For starters, Telstra is only responsible for the cable from the outside of the house to the exchange, not for the internal wiring inside the house. Much of the time wasted in fault finding is in first determining whether the issue is internal or external. None of this is solved by FTTN. The dodgy copper is still there and still prone to faults and dropouts.”
Addressing the issues
It’s very tricky to gauge the exact situation. It’s almost as easy to believe that the network is rotten with good patches as it is to believe that it’s good with rotten patches. For the sake of brevity we’ve left out many of the other reports, which overwhelmingly back up the “rooted” claims above, but you can read the key reports and experiences here. While the experiences are from disparate people, the details and the same themes keep appearing time and time again and they are worth reading. However, it is this particular tale has become something of a cause celebre.
There are unquestionably sections of the network that are not suited to upgrades and the Coalition acknowledges this – the worst-affected areas will get FTTP “Where it makes economic sense.” But with every single home having a different set of conditions compared to its neighbour, having a few houses on a street with copper that can’t handle FTTN is unlikely to be enough to warrant a whole neighbourhood getting FTTP. At some point a threshold will need to be established whereby a neighbourhood will only receive FTTP when a certain proportion of bad homes versus good homes is reached.
The Coalition makes a sideways mention of remediation in its Background Notes: suggesting that nine per cent of premises will be in areas too bad to be upgraded and so they will continue to get FTTP:
Excluding MDUs, about 9 per cent of premises are connected using FTTP in the period 2014-2019. These premises are assumed to be in areas with the poorest quality or most maintenance]intensive copper networks.
The obvious problem is that if more than nine per cent of the network is rotten – and if the distribution is dispersed rather than in tightly-defined areas – the Coalition’s model will not work and cost blowouts would be guaranteed.
An additional headache is that copper condition directly affects the promised minimum download speeds of 25Mb/s by 2016. For instance, in the UK, corrosion has seen average speeds drop from 53Mb/s to just 6Mb/s. Furthermore, to hit the promise of a minimum 50Mb/s for all Australians by 2019, following German upgrade paths, the copper would need to be remediated for FTTN and then remediated again to add the subsequent “Vectoring” speed boost. One commentator puts the latter cost at an additional $20bn. There’s also no word on who would fix problems that occur with the copper within people’s own premises.
It’s also worth adding that neither the UK nor Germany report widespread problems with the networks due to plastic bags, outside plant records, tropical climates and general neglect to the levels that we see in Australia.
The key problem is that the copper network represents the cornerstone of the Coalition’s $30bn policy and is closely-related to the current $11bn deal between NBNco and Telstra. Can it be justified for vast sums of public money to be spent procuring a network, with such a history of neglect, without it being audited first?
This, in turn, raises the question of practicality: the condition of every length of copper is different and so you couldn’t gauge the network’s condition based on any localized examination. The only way to audit each line is to send engineers to each pillar and home to manually performa a test but there are over eight million lines in service.
Furthermore, if Telstra were to become central to a broadband rollout based upon its copper, how would work practices change? Would every work order for fault fixing, remediation and upgrading be done to a high standard? What about the backlog of work and numerous plastic-bag fixes that stem from a decade of neglect? Can this all be addressed in time to reach the goal of enhanced broadband for all in just three years? If so, who is going to pay for it? Malcolm Turnbull says Telstra won’t charge any extra. Is that conceivable without significant additional incentives being offered?
Whatever the reality, the actual cost of using the copper is directly linked to its condition. The question now is, just how damning is the evidence? Does it reflect a noisy minority or the tip of the iceberg and does anyone, even Telstra, really know for sure?