An SDN vulnerability forced OpenDaylight to focus on security

Open-source software projects are often well intended, but security can take a back seat to making the code work.

OpenDaylight, the multivendor software-defined networking (SDN) project, learned that the hard way last August after a critical vulnerability was found in its platform.

It took until December for the flaw, called Netdump, to get patched, a gap in time exacerbated by the fact that the project didn’t yet have a dedicated security team. After he tried and failed to get in touch with OpenDaylight, the finder of the vulnerability, Gregory Pickett, posted it on Bugtraq, a popular mailing list for security flaws.

INSIDER: 5 ways to prepare for Internet of Things security threats
Although OpenDaylight is still in the early stages and generally isn’t used in production environments, the situation highlighted the need to put a security response process in place.

“It’s actually a surprisingly common problem with open-source projects,” said David Jorm, a product security engineer with IIX who formed OpenDaylight’s security response team. “If there are not people with a strong security background, it’s very common that they won’t think about providing a mechanism for reporting vulnerabilities.”

The OpenDaylight project was launched in April 2013 and is supported by vendors including Cisco Systems, IBM, Microsoft, Ericsson and VMware. The aim is to develop networking products that remove some of the manual fiddling that administrators still need to do with controllers and switches.

Having a common foundation for those products would help with compatibility, as enterprises often use a variety of networking equipment from many vendors.

Security will be an integral component of SDN, since a flaw could have devastating consequences. By compromising an SDN controller—a critical component that tells switches how data packets should be forwarded—an attacker would have control over the entire network, Jorm said.

“It’s a really high value target to go after,” Jorm said.
The Netdump flaw kicked OpenDaylight into action, and now there is a security team in place from a range of vendors who represent different projects within OpenDaylight, Jorm said.

OpenDaylight’s technical steering committee also recently approved a detailed security response process modeled on one used by the OpenStack Foundation, Jorm said.

If a vulnerability is reported privately and not publicly disclosed, some OpenDaylight stakeholders—even those who do not have a member on the security team—will get pre-notification so they have a chance to develop a patch, Jorm said. That kind of disclosure is rare, though it is becoming more common with open-source projects.

The idea is that once a flaw is disclosed, vendors will generally be on the same page and release a patch around the same time, Jorm said.

OpenDaylight’s security response process is “quite well ironed out now,” Jorm said.

---

Best Microsoft MCTS Certification, Microsoft MCITP Training at certkingdom.com

Future-proof your IT career: 8 tech areas that will still be hot in 2020

It’s prudent for IT pros to cultivate skills that are in high demand. Even better are skills that will stay in demand. Here are eight key technology areas that show no signs of falling out of favor.

Wanted: Programmers, security experts, cloud capacity managers
More than 90% of U.S. companies are using some form of cloud computing, according to CompTIA's most recent Trends in Cloud Computing study. Moreover, the November 2014 report found that companies are moving infrastructure or applications between private and public clouds. IT leaders predict that movement will accelerate in the future, which will generate a host of cloud-centric jobs, including cloud security.

A related position will be dedicated to cloud capacity management. "We expect many [organizations] will operate in a hybrid environment, a mix of private and public cloud, so the question becomes how to dynamically switch demand for compute and storage from private and public clouds," says Mike Sutcliff, group chief executive for Accenture Digital. "That's going to require new techniques and disciplines that many IT organizations don't have in place today."

Programmers skilled in Perl, Ruby, Ruby on Rails and Python, Java and JavaScript, as well as those comfortable with API development and a DevOps environment, will also be in high demand, because cloud technology depends heavily on those disciplines. (See 10 hot cloud computing skills for details.)

Wanted: Data architects, integration experts, Hadoop pros
Cliff Justice, leader of KPMG's Shared Services and Outsourcing Advisory practice, says organizational needs around analytics will be huge, driven partly by the sheer volume of data collected but also by the increasing number of applications (such as robotics) fueled by analytical output. As a result, companies are adding and creating IT positions to handle the work.

According to Barry Brunsman, principal in KPMG's CIO Advisory Management Consulting practice (pdf), you'll see roles like these: Data architects, who design the structure to support emerging needs; data integration engineers, who ensure that data solutions and analytics from any number of sources can be integrated; and IT planning analysts, who aggregate and analyze data from many internal and external sources to help IT know what its business partners are likely to need in the future.

Technical titles that are and will remain hot include Hadoop developer, data engineer, big data software architect and enterprise data architect, says Christian P. Hagen, a partner with the Strategic IT Practice at management consulting firm A.T. Kearney.

At the same time, organizational demands around analytics will create a new batch of leadership positions tasked with understanding how to use analytics to achieve goals and objectives. "Analytics won't mean just working with tools. Companies will need someone out in front, someone who can get at how analytics will transform the company and IT as well," Hagen says.

Hagen says leadership positions emerging in this field are chief analytics officer, chief data officer, chief digital officer, head of business analytics and vice president of enterprise data.

Wanted: "Digital artisans"
The pressure to be more than a pure technologist will continue in the upcoming years - and that means more than adding one or two business skills to your resume. Tech pros who successfully navigate the changes roiling the industry will be able to demonstrate business acumen across the spectrum, says R "Ray" Wang, founder and principal analyst with Constellation Research Inc. He calls these new specialists "digital artisans," explaining that they're "those who can balance right brain and left brain skills."

Middle-of-the-road products, services and solutions aren't enough to sustain companies in an increasingly competitive landscape, Wang says. To thrive in the next five to 10 years, organizations need to seek out talent "that can think outside of the box but execute within the system," he says. To deliver that kind of strategic value, IT pros need to be authentic, relevant, transformation-minded, intelligent, speedy, artistic and non-conformist. (Get it? A-R-T-I-S-A-N.)

Wanted: Hardware, software, analytics experts
The 2014 PwC report The Wearable Future (pdf) sees a world where wearable devices will be used to train new employees, speed up the sales process, improve customer service, create hands-free guidance for workers and improve the accuracy of information collected to serve the growing analytics movement at companies everywhere.

Jack Cullen, president of IT staffing firm Modis, predicts the move to wearables could spur as much, if not more, new development as did the move to smartphones. "By the time 2020 rolls around, wearable devices could be as common as the iPhone today, and that creates all new opportunities," Cullen says.

Cullen expects that organizations of all kinds will identify workers and processes that could benefit from wearables, which it turn means IT departments will seek out technologists with the ability to deploy, manage and maintain hardware as well as experts who can develop, customize and support the applications and analytics programs that will make wearables useful within their specific organizations.

Wanted: In-the-weeds tinkerers and big-picture thinkers
Research firm IDC predicts in its Worldwide and Regional Internet of Things 2014-2020 Forecast that the global IoT market will grow from $1.9 trillion in 2013 to $7.1 trillion in 2020.

"Technology is being built into almost everything we have," says David Dodd, vice president of IT and CIO at Stevens Institute of Technology. That means a bright future for technologists who understand the underpinnings of this kind of connectivity. Indeed, IoT could breed a new specialist who can combine skills in hardware, engineering, programming, analytics, privacy and security.

Dodd, though, believes the IoT skill most in demand will be in understanding what value comes from all this connectivity. Organizations are realizing it's not enough to simply connect items and gather data, they need to know how those connections and the data they generate can solve problems or advance organizational goals. Companies "want people who can understand and formulate the future of IoT," he says.

Position yourself for long-term growth
Smart companies have a corporate roadmap that spells out where they'd like to be three, five and 10 years out, how they're going to get there, and how technology fits into that vision. As a smart IT professional, can you say how your skills and position figure into your company's plans -- or the industry's as a whole?

Sure, organizations will still need programmers and developers, but they'll want (and pay better salaries to) programmers who know how to work with robots and developers who know how to apply their craft to wearable devices. So, yes, while labor market experts expect that IT as a whole will continue to add good jobs through 2020 and beyond, savvy tech pros are taking pains to ensure their personal roadmap is steering them towards concentrations with maximum longevity.

What follows are some specialties worth pursuing to future-proof your tech career.

Wanted: Tech experts to lay the groundwork for enterprise AI/robotics
Artificial intelligence and robotics have already moved from science fiction to reality, and soon they'll be coming to a business near you. According to a 2014 Pew Research Center report (pdf), these technologies "will permeate wide segments of daily life by 2025, with huge implications for a range of industries such as healthcare, transportation and logistics, customer service and home maintenance."

Not surprisingly, technologists skilled in this area will be in high demand, says KPMG's Justice. He notes that IT professionals will have roles to play in programming, integrating and building out the infrastructure for organizational applications of AI and robotics.

Wanted: Programmers to tap internal, external power of APIs
There's already plenty of buzz around application program interfaces (APIs) -- the sets of routines, protocols and tools that specify how software components should interact and facilitate access to Web-based applications.

Software vendors have been providing API for years, and now companies of all disciplines are making theirs public so other developers can design applications that interact with their original software. For that reason, the importance of APIs is about to explode. Companies will require more and more APIs to tap the power of emerging technologies, such as the Internet of Things, robotics and artificial intelligence, as well as maximize value for existing tech-driven trends such as mobile connectivity.

IT shops will need professionals to actively develop and manage APIs for use within the organization and to connect with outside users, Accenture's Sutcliff says. These technologists need to have strong development skills as well as an understanding of data sources, data structures and the organization's applications portfolios. Sutcliff notes that this position won't be about one specific language or API, but more about assembling pieces together.

Wanted: Broad and deep security chops
The U.S. Bureau of Labor Statistics anticipates a 37% growth in information security analyst positions between 2012 and 2022 for good reason -- all these emerging technologies are requiring, and will continue to demand, even more attention from an organization's security program.

"For all the great opportunities that social and mobile and cloud and analytics and the Internet of Things are going to bring, any economic gains that will be realized by all these new technologies can be undercut significantly if there aren't really robust security programs and protocols in place," says Matt Aiello, a partner in the Washington office of Heidrick & Struggles, which specializes in recruiting CIOs and senior-level technology, engineering and operations executives. Aiello and others say the security expert of the future will need to ensure that security is embedded in all levels.
 

---

Best Microsoft MTA Certification, Microsoft MCTS Training at certkingdom.com

20 epic Microsoft Windows Automatic Update meltdowns

20 Windows Automatic Updates from hell

Fifteen years ago, Microsoft introduced automatic updating to the unwashed Windows masses. Fifteen years later, it’s hard to find a Windows user who hasn’t bumped into at least one problem with a Windows update or knows someone who has. That’s a billion and a half people.

From inscrutable driver problems to bricked machines and everywhere in between, Automatic Update is a poster child in “what’s wrong with Windows” circles -- rightfully so.

Hope springs eternal that Windows 10 will finally bring relief, but much depends on the determination and deep pockets of Those in Charge. One thing’s for sure: In the land of Win10 milk and honey, customers don’t want to be treated like cannon fodder.

Here’s my take on the 20 worst Microsoft Automatic Update patches of all time. Based on either the amount of pain inflicted or the number of people afflicted -- or both -- they deserve their notoriety.

Those who cannot remember the past are condemned to repeat it.

November 2001: The UPnP patch debacle
Microsoft introduced Windows automatic updating as one of the great new benefits in Windows Me, around September 2000. A year later, we were treated to a Keystone Kops episode in the guise of MS01-059 -- ostensibly, a patch to the Windows Universal Plug 'n Play subsystem that prevented a buffer overrun. In fact, I think it was the first (though hardly the last) security bulletin conceived and scripted by Comedy Central.

Microsoft patched, repatched, and re-repatched the patch. The FBI's National Infrastructure Protection Center followed along like a kid cleaning up after his dog: NIPC issued a warning about the security hole, an update, another update, and ultimately an advisory that Microsoft had finally solved the problem.

April 2004: Windows 2000 bricked
In April 2004, Microsoft sent a slew of patches down the automatic update chute, one of which (MS04-014) locked up a sizable percentage of all Windows 2000 machines. That patch was supposed to fix a hole in the Jet Database Engine.

Knowledge Base article 841382 tells the tale:

[Y]ou may experience any one of the following symptoms:

• Your computer appears to stop responding at startup.
• You cannot log on to Windows.
• Your CPU usage for the System process approaches 100 percent.

The company sure plugged that one.

April 2006: The pretax predicament
On Black Tuesday in April 2006, Microsoft released MS06-015, a patch for Windows Explorer. By the weekend, most Windows users with Automatic Update turned on got it -- right across the face. The weekend before tax day, many Windows customers found they couldn't navigate to the Documents or Pictures folder, couldn't open or save files, had to type http:// into Internet Explorer to keep it from freezing, and much more.

We ultimately discovered that the patch messed up any machine with an older HP scanner program or an older Nvidia video driver.

Microsoft's ultimate workaround (KB 918165) included a manual fix procedure that any computer-science grad would be proud to explain, if they can figure it out.

April 2006: Windows Genuine Spyware -- er, Advantage
Microsoft uses the Automatic Update channel (and permissions) to install Black Tuesday security patches, as well as non-security-related patches. My favorite example came in late April 2006, when somebody at Microsoft decided Automatic Update would be a great way to install the new Windows Genuine Advantage, uh, feature.

That version of WGA (in addition to throwing off bogus "not genuine" messages) installed a component called WGA Notification that phoned home -- sent information to Microsoft about the current computer -- with absolutely no notification to or approval from the customer. Lawsuits ensued. I called it Windows Genuine Spyware.

August 2006: The IE patch that created a new buffer overflow hole in IE
Let's hear it for MS06-042, the cumulative security update for Internet Explorer that not only caused IE to crash, but also introduced a security hole of its very own.

In late August, Microsoft owned up to problems in KB 923762: the part where IE6 crashes while looking at a valid website. Solution? Install the latest, greatest version of MS06-042.

Then in September, Microsoft had to reissue the patch again to "address a vulnerability documented in the Vulnerability Details section as Long URL Buffer Overflow -- CVE-2006-3873."

KB 918899 lists 15 separately identified problems with this patch, from crashes to freezes to inexplicable behavior.

December 2007: Internet Explorer crashes on sites with lots of graphics -- like msn.com
Yet another cumulative security update for IE, MS07-069 patched IE so well that many WinXP SP2 customers reported IE6 freezes on sites with many graphics. If you had automatic updates turned on and were running plain-vanilla WinXP SP2, after the patch was installed, you couldn't let IE go to the default IE6 home page, msn.com.

If you installed the patch for Internet Explorer 7, your (third-party) firewall might not have recognized IE. As a result, it may have kept IE from going out to the Internet. IE produced the marvelously informative error message "Webpage cannot be displayed."

It took weeks, but Microsoft finally acknowledged the problem and posted a downloadable fix program in KB 946627.

April 2008: Quicken suddenly stops working
Nobody seems to know why, but Microsoft suddenly released the .Net 2.0 Service Pack 1 on a Thursday, one week before tax time, via the automatic update chute. The patch itself had been available as an optional, manual download for months, but somebody flipped the auto update switch.

Within minutes, Quicken users were complaining. QuickBooks got hit, as did TurboTax and software from Commerce Clearing House.

How bad was it? If you were bit, uninstalling, then reinstalling QuickBooks didn't solve the problem. You had to uninstall, then reinstall .Net 2.0 -- if you could get it to uninstall.

All through 2009, 2010, 2011: Bad .Net patches
Over and over again, we saw botched .Net patches -- some refused to install, others left .Net dead, others clobbered programs that relied on .Net. It started in January 2009 with a patch that claimed to push .Net Framework 3.5 to Service Pack 1, but didn't.

Another patch, in March 2009, also identified as .Net Framework 3.5 SP1, installed .Net Framework 2.0 SP2 and .Net Framework 3.0 SP2 as well. It was an unholy mess that had us going in circles for months.

We saw many more .Net patching problems in 2010 and 2011, all compliments of Automatic Update.

March 2009: The XP AutoRun blocker that didn’t
It took Microsoft forever to post a patch that disabled AutoRun in Windows XP. AutoRun, indicted as the culprit behind mass Conficker infections, deserved to die, but Microsoft's first and second attempts to talk people through the disabling procedure didn't work.

The final solution is so incredibly convoluted that pages of KB 967715 are devoted to explaining the interactions of all the patches, both delivered via automatic update and manually downloaded. It's complicated. Bottom line: If you installed only one automatic update, you might've thought that you fixed AutoRun, but you didn't. It took several patches over several months to finally get it right.

December 2010: Patch brings down Task Scheduler
MS10-092 was an innocuous patch, designed to plug a hole in Windows Task Scheduler.

But shortly after people started installing it, they saw messages saying, "The task image is corrupt or has been tampered with." In some cases, the task was killed. In other cases, the machine froze. Simply uninstalling the patch didn't solve the problem -- great prelude to the holiday season.

KB 2305420 has pages and pages of manual workarounds.

January 2011: A reliability update that wasn’t
On January's Black Tuesday, Microsoft pushed a nonsecurity patch into the Automatic Update black hole. Known as KB 2454826, Microsoft claimed it was a "performance and functionality update." Details about the patch at the time were sketchy, but the 0x7F blue screen crashes weren't.

Microsoft's advice: Manually uninstall the patch. That's your reward for turning automatic updates on, bucko.

It wasn't until the next month that we discovered the real reason why Microsoft pushed this nonsecurity patch out the Black Tuesday chute: It's a prerequisite for installing the Internet Explorer 9 Release Candidate, which Microsoft was flaunting at the time.

April 2012: TurboTax won’t print
Just before tax day -- tell me if this is starting to sound familiar -- Microsoft released MS12-025, yet another botched .Net patch.

(For the sake of brevity, I didn't bother to list separately MS10-070, MS11-039, MS11-044, MS11-066, or MS11-069, all of which were incredibly botched .Net patches.)

This particular patch kept TurboTax from printing tax forms ... on tax day. #epicfail

May 8, 2012: Duqu patch installation failure
A massive patch known as MS12-034 (with many associated KB numbers) left some Windows customers who used Automatic Update wondering what had gone wrong. Some found that the installer failed with an Error Code 0x8007F0F4. When they checked the KB 2686509 support article, they were instructed to delete a keyboard log file. Many people couldn't find the file.

The instructions in KB 2686509 go on for pages, explaining how to modify and move keyboard layout files -- in response to a known, anticipated error thrown off by the installer. Microsoft finally got around to creating a Fix it that made the patching easier. But lots of unsuspecting Windows consumers wasted hours trying to make heads from tails out of this automatically updated disaster.

February 2013: Blue screens on Internet Explorer 9
Once again, Microsoft threw a bunch of machines into a tizzy by releasing a nonsecurity patch on the fourth Tuesday of the month -- and sending it down the Automatic Update chute.

This time, KB 2670838, a "Platform Update for Windows 7 x64-Edition" messed with IE9 so badly that it would put a black bar on the right side of the screen. Click on the bar, and your PC died with a blue screen.

Fortunately, the fix is to uninstall the bad patch.

April 2013: More blue screens
This time, MS13-036/KB 2823324 -- a Black Tuesday security patch designed to replace a kernel-mode driver -- triggered all sorts of bogus warnings and frequently froze machines. Primary suspects include a common IE add-in from Brazil and Kaspersky Antivirus.

Microsoft pulled the patch, then issued a replacement patch: "Microsoft has released security update 2840149. This security update resolves the issue that was introduced by security update 2823324."

August 2013: The biggest, baddest bungled batch ever
Within 48 hours of the month's automatic update, Microsoft publicly admitted six Windows patches were bad and pulled four of them, all associated with MS13-066 and Active Directory Federation Services.

As far as I can tell, that's a record. It's not only a record for bad patches. It's a record for how quickly Microsoft acknowledged, documented, and in some cases, pulled the offending patches. We’ve seen bad Patch Tuesdays since, but this one stands out, in both good and bad ways.

November 2013: Outlook 2013 gets special treatment
One of the patches in the November 2013 set caused no end of problems with Outlook 2013 -- Outlook hangs when trying to sync IMAP accounts; Out of Office replies on Exchange Server triggered "currently unavailable" messages; Free/Busy data for the Outlook Calendar didn't download; S/MIME certificates wouldn't validate; and more.

Unfortunately a second patch released in the November crop made it impossible to fix all of those Outlook 2013 problems by simply uninstalling the bad patch. In the end, users in the know discovered they could resuscitate Outlook 2013 by uninstalling both patches, then deleting and rebuilding the Outlook profile.

Microsoft, which did so well in August -- “well” in the sense it cleaned up quickly -- really blew it in November.

May 2014: Windows 8.1 Update won’t install, Microsoft backs off its deadline
In a scene straight out of Dante’s "Inferno," Microsoft cracked the whip and told all Windows 8.1 users that they had to install the KB 2919355 update (so-called Win8.1 Update 1) by May 13, or they wouldn’t get any new patches. Predictably and with much wailing, a vocal subset of Windows 8.1 customers discovered Update 1 wouldn’t install, for love nor money -- or anything resembling either or both.

Quite dramatically (tell me if you can visualize the seventh ring), Microsoft finally relented on May 12 and said it would allow the tardy minority to receive updates -- but only this one last time.

(I think it’s poetic justice that Win 8.1 Update 2 stalled, then fizzled completely, ultimately leading to a re-release that didn’t do much.)

August 2014: Blue screens all around, Microsoft recommends you manually yank the patches
Four patches in August were credited with driving blue screens on Windows 7, 8, 8.1, and RT machines. Microsoft pulled the patches on Sunday, then issued a very unusual notice, buried deep in a Knowledge Base article: Even if you weren’t having any problem with the patches, you were supposed to manually uninstall them.

Apparently the patches continue to cause problems, even after they were installed, in certain unusual circumstances.

It’s a bit much to tell your Aunt Mabel to manually uninstall a handful of patches, based on a warning in a KB article, but there you have it.

December 2014: Roughly a quarter of all the patches this month generated problems
With only a few of the bad patches fixed before the end of the year, December 2014 represents the worst combination of bad patches and lackadaisical responses I can recall. In response to the unprecedented number of screw-ups, Microsoft pulled a few patches, released two Fix its, created a Silver Bullet patch that specifically killed one of the bad patches, and wrote up numerous manual work-arounds.

Even at this late date, more than two months later, the problems brought down on Excel macro programmers haven’t been fixed.

Would you say Automatic Update is getting better?

Come out of the Automatic Update cave and into the light
That's by no means an exhaustive list. Some problems are inevitable when you're dealing with a Windows hardware and software gene pool that looks like the La Brea Tar Pit, but I think you can draw three important conclusions:

First, patching Windows is hard.

Second, Microsoft needs to do a better job of tracking and reporting on problems as they appear.

Third, for Pete's sake, set Automatic Update to Notify but Don't Download on any machine controlled by a reasonably savvy Windows jockey.

If somebody tells you differently, point them to this list. If they're still convinced Automatic Update is the way to go, ask them to refrain from dragging their knuckles on the floor.

---

Best Microsoft MCTS Certification, Microsoft MCITP Training at certkingdom.com