Monitis: Cloud Monitoring Blog

Guest Post:

Hello I’m Rick Stivers, and I’m the Director of Network & IT Services for Schoolwires. Schoolwires provides strategic online communication, community-management and productivity solutions to the K-12 education market. Our core product is Centricity, which brings together robust and flexible website management, community management and web 2.0/social network capabilities in a single, user-centric solution.  We also deliver Schoolwires Synergy™, a digital file sharing solution, and Schoolwires Assist™, a service request solution. We currently work with nearly 1,000 districts, over 5,000 schools and over  5 million users across the nation.

 We currently provide four different hosting models.

- ASP Hosted at our Managed Facility through Expedient Communications

- Educational Service Agency Hosted at their own facilities which serve districts in their regions.

- Client Hosted – Self Hosted at School Districts on their own equipment

- Schoolwires Appliance – Schoolwires maintained equipment at School Districts.

Over the past year we have really pushed toward the cloud. We implemented an Enterprise VMware Architecture with High Availability, DRS, and VMotion. One benefit of this push was higher reliability on the hardware side and reduction of our server footprint.

We began working with Monitis roughly two years ago. At the time I was unhappy with the current solution I inherited. It was a solution which required our own server, and was not robust enough for the number of sites we needed to monitor. We did an exhausting search and were just not happy with what we were finding due to the number of sites we needed to monitor and the ROI for many solutions were not cost effective. I found Mon.it.or.us, and immediately liked what I saw during the trial process. I really liked the fact that they provided a solution along the same model as ours (SaaS), and that it was priced very competitively. It could also scale to the number of sites we currently had (more than 1200) and where we expected to be in 3 years. The interface was also very user friendly and with the AJAX interfaces felt more like a desktop application than a web tool. The reports are very helpful, and really allow us to focus our attention on the lower performing sites. Monitis allows easy exporting of raw data. This has helped with productivity, because I can easily categorize reports based on hosting type and really pinpoint issues related to the location. I also liked that they were innovating and really willing to listen to feedback on feature requests. One of the real benefits of their solution is that it gives an independent view on our sites SLA numbers. Unlike software we load internally on our network and which increases our cost because it’s another server and application we need to manage.

Part of my job description requires that our sites maintain a certain SLA, and without Monitis this would be difficult to track and maintain. The downtime alerts are very accurate and allow us to respond quickly to service outages. We really value this partnership and will continue to work with Monitis to help improve their product and services. Thank you…

Rick Stivers, the Director of Network & IT Services for Schoolwires

So, you’ve migrated to the cloud, or at least you’re considering it, and now you want to employ a monitoring solution. Good decision.

After all, enterprise networks are getting more complex as globalization continues to rage – despite the recession. And outsourcing and the growth of wireless are expanding networks beyond their traditional reach. Consider, too, that IT departments are being asked to guarantee that applications and services run well across both private and public networks.

Beyond that, the rise and growth of virtualization and cloud computing makes the job of monitoring and troubleshooting all the components of a computing network even more of a challenge. Why? Because virtualization removes the hardware from the application and creates traffic more difficult to monitor and troubleshoot. And with cloud computing, apps are not housed on the existing enterprise infrastructure and instead reside over the public network. Because ownership of the issue is now shared between the IT staff and your cloud service provider, it makes it even more difficult to correctly identify and track down performance issues.

So the heat is on; and ratchets the pressure up a notch with CIOs demanding cost reduction and higher service levels. All in all, it means finding ways to work more efficiently and effectively. And the right network management tools will offer you the right mix of products that meet both your monitoring and management needs.

So what kinds of capabilities should you look for in a network management system? In this post, I’ll focus on the generalities, and then in subsequent posts, I’ll discuss specific vendors that might be useful to you.

First, because change is a constant, not only in business but in life, too, IT must keep the network up and running regardless of moves, additions and changes. While network management solutions were once built solely to manage devices, now they’re more complex because they have to manage new types of traffic – such as voice and streaming video which are very latency sensitive. Plus, now that private corporate networks are on the public Internet, there are new concerns that IT must be concerned about – such as traffic levels and security issues.

So, consequently, it’s important to consider solutions that can monitor:

- network traffic

- end-user activities

- applications

- networking protocols

- servers

- network hardware

- virtual machines

- protocols

- cloud platforms

You need these elements for a complete end-to-end picture of network health.

These individual elements are necessary for the whole because, too often, when there are performance problems, the network is the first to be blamed. In fact, the problem could just as easily be an application glitch or trouble with end-user activities. Monitoring provides IT with a more proactive rather than reactive approach – as it will help mitigate problems before they happen and reduce mean time between failures.

Right now, unfortunately, most enterprise IT network departments use many disparate tools to try and improve visibility. Plus, IT departments are too often structured into distinct organizations, and that only compounds the problem because information isn’t shared. For greater visibility, things must change, and IT groups must work to share information in order to be proactive.

Network management suites have expanded well beyond monitoring just devices, and this broader view helps networking engineers determine potential bottlenecks before they happen. Putting the right enterprise network management solution in place sheds light on the entire network and improves overall network stability and reliability.

At Monitis, our end-to-end monitoring system offers:

- True 1 min frequency monitoring simultaneously from multiple locations in America, Europe, Asia and Australia

- The ability to customize and set up your own monitoring locations

- No false alarms – failures verified across multiple locations

- Monitors websites, e-Mail servers, firewalls, VoIP, databases, Domain Name Servers, routers, Web Servers from end user perspective

- Supported protocols – HTTP, HTTPS, FTP, SMTP, POP3, IMAP, SSH, PING, TCP, UDP, SIP, MySQL, DNS.

- Internal server and network monitoring including CPU, Memory, Server Processes, Bandwidth, SNMP, telnet.

- Web page content check – specify string to be checked for existence/non-existence on web page

- WebMap view – see all your servers and web sites in the single map view

I love to read propaganda. It’s so entertaining.

Xinhuanet, the top news site in China, reports that “most Chinese Internet users believe they will be OK with a no-Google Internet despite all predictable inconvenience.”

So how does Xinhuanet know that? The site cites a survey by the Global Times newspaper, an affiliate of the state-owned People’s Daily, in which 27,000 Internet users were asked “What’s your opinion of Google pulling out of China?” About 84% said they “don’t care.”

Just read how the government newspaper described why Google decided to re-route users to a search site in Hong Kong that it isn’t censoring (on behalf of the government):

“Google stirred up controversy in the world’s media and on the Internet in January when the company’s chief legal officer, David Drummond, said in a blog that Google might shut down google.cn and its China office due to disputes with the Chinese government and unidentified cyber attacks against its Chinese users.”

You can really tell a lot about the slant of a news story by the language used…”stirred up controversy”…”due to disputes with the Chinese government.” How about discussing why Google’s quitting its search business there: the censorship issue?

It remains to be seen now if China will block citizens’ access to the Google Hong Kong site. For now, it isn’t, according to the The Wall Street Journal.

The way I see it, Google, after threatening to leave China over censorship issues, finally took action because of a cyber attack on Chinese dissident Gmail users – an attack believed to have come from China. That was the last straw for Google.

Why do I care so much about this? The protector in me can relate to Google’s actions. As the owner of a monitoring company, part of my mission is to warn clients of hackers and other threats. You don’t want to let your clients down.

Going head to head with Microsoft, The Wall Street Journal reports that Google is preparing to launch an online store that sells business software integrating with its cloud services. Google will apparently hire third-party software companies to design business software which it will then integrate with and enhance Google Apps.

The Journal quotes unnamed sources as saying this is Google’s plan and that the store, a new and improved version of its Solutions Marketplace, will come online in March 2010. Customers would be allowed to buy Google’s partners’ software on the site, and users will be able to immediately access or use their applications – via the Gmail or Google Docs menus.

What are some of the improvements to Apps that Google’s looking at? Think tinkering that produces enhanced security features, the ability to import contacts, tools to access online files offline. Officially, Google is not talking about this yet: The Journal quoted a Google spokesman as saying that the company is “constantly working with our partners to deliver more solutions to businesses, but we have nothing to announce at this time.”

The new store is seen as part of the company’s business strategy to grow its online software business – and to get customers away from Microsoft programs. After all, Google can’t rely on advertising revenue forever.

But one consequence of this new strategy of relying on outside software developers is that there could be quality issues. I hate to even think of anything negative. But, indeed, concern about storing data online is why many large enterprises are fearful of migrating from internally housed Microsoft applications to cloud-based ones.

So, this could be another reason why it’ll become more important than ever for companies using Google’s suite of software (or Azure or any other cloud platform, for that matter) to employ back-up plans to monitor the reliability of their cloud services and help keep their data available and safe.

Every day we read more about local, national and international governments migrating to the cloud for data storage and applications use. Los Angeles is a well-known example of a city that switched from internally hosted email to the cloud’s Gmail app. Usually, the reasons are to achieve more efficiencies and savings and to give citizens more and faster access to services and information.

But there is another level of discussion around the cloud that government and the private sector is increasingly engaging in. It’s a more strategic, broader view of cloud issues, for example, regarding security standards.

Recently, Microsoft hosted a stimulating roundtable at the Aspen Institute in Washington about cloud computing, and both government representatives and cloud players got a chance to speak their minds – and have dinner, too – according to a Washington Post article.

Microsoft general counsel Brad Smith spoke about the opportunities that cloud computing presents to them and other cloud service providers. But it’s putting greater pressure on Washington policy makers and Congress to protect companies from attacks on their servers and privacy. Did you know that law enforcement routinely ask for data on the cloud – which is NOT protected from existing privacy protections? Microsoft is clear on what it wants the government to do: update the Electronic Communications Privacy Act to apply to protections on the Web. It also wants stronger rules against cyber attacks by reforming the Computer Fraud and Abuse Act.

“Courts are less clear about what is reasonable expectations of privacy when it comes to data that goes to a third-party” server, Smith said at the dinner, according to the Post. Apparently, many in the room said there should be basic privacy and security protections for all consumers.

And the Federal Trade Commission is investigating privacy concerns on the cloud, as end users often aren’t aware of where their data is being held or what kind of protections are in force.

And then there’s the issue of information borders on the cloud. What kind of rights should a cloud app user who’s living in the U.S. expect when their information is hacked by users in China? “This is really popping and something that will be very disruptive,” said Alec Ross, senior adviser on innovation to Secretary of State Hillary Rodham Clinton, in the Post article. “We need to think about who the great actors are and what they do with that information … and then set forth a clear set of values and norms.”

What would I have said if I were in the room, part of the dinner? Something like this: while we wait for the perfect cloud world to arrive, where everyone’s data is safer than Fort Knox, there are ways that businesses can add a little insurance to using the cloud, such as monitoring your cloud provider for reliability, or monitoring your sites and transaction volume. With warnings delivered in a variety of ways, you get some peace of mind that whatever trouble is brewing can be headed off.

There’s a new report out from a very well-respected environmental authority, Greenpeace, that paints a dark picture for cloud computing pollution. I thought it was so interesting because when a lot of people talk about cloud computing they can make erroneous assumptions that the cloud is carbon-free. This comes from the notion that by not having to run and maintain a slew of internal servers, companies reduce their energy consumption – and thus emissions.

Well, the Greenpeace study, “Make it Green: Cloud Computing and its Contribution to Climate Change,” says quite the opposite. The study, which dubs 2010 as “The Year of the Cloud” due to the growth of new cloud innovations such as e-readers Kindle and the iPad (not to mention the rising number of enterprises migrating to the cloud to do business), says cloud computing GHG emissions are set to triple by 2020. As more people buy and use Kindle and iPad, there’ll be more need for cloud data centers, and thus more power needed to run them.

Specifically, the report finds that, “at current growth rates, data centers and telecommunication networks, the two key components of the cloud, will consume about 1,963 billion kilowatts hours of electricity in 2020, more than triple their current consumption and over half the current electricity consumption of the United States. That’s more than France, Germany, Canada and Brazil combined.

Further, Greenpeace urges the IT industry to convert to renewable energy when possible, for example, solar or wind-powered energy. In a move in the opposite direction, Facebook earlier this year opened a new data center in Oregon, powered by a coal plant in Idaho.

To me, it’s kind of a “Catch-22″ situation when you talk about carbon emissions from the cloud. Yes, there’s definitely a need to explore and adopt alternative energy sources to run data centers, but think of all the trees we’re not cutting down by more people reading books on Kindle. Or, what about all those planes grounded because people are now meeting via SaaS virtual meetings programs? And, yes, companies, especially large ones, are saving by not running cooling systems to keep their servers maintained.

Interested in how much your enterprise is contributing to cloud computing pollution? Well, cloud monitoring reports, for example, number of instances and performance trends, can give you some idea of the resources cloud platforms and services are using to support your business.

A recent article I read on cloud computing risk asks a good, basic question that I think every company should consider when weighing the benefits of migrating to the cloud: cloud computing: Are the advantages of Cloud computing compared to in-house management of data worth the risks or not?

This writer was referring to the security breaches and data privacy violations that have plagued cloud providers, for example, the recent breach of Gmail by Chinese hackers that we aimed at that country’s political dissidents.

Sure, your company can figure out the IT cost savings and efficiencies inherent in switching to cloud computing, and like many other companies, you may even hit your financial goals. But what “cannot be put into the financial statement or projected costs savings are the unknowns in your company’s brand protection should something go wrong,” the writer says. “Other considerations are litigation, insurance (risk) costs, and other service liability claims that could arise.”

The simple fact is that the more information about or from you stored in central repositories and accessed from single-source environments, the more risk that cloud computing suppliers assume. Add to that the significant pressures on companies today to cut their IT costs, determining how much risk a company undertakes managing information and intellectual property.

What kinds of risk do companies assume when migrating to the cloud? Data centers operated by third parties “invite more than just criminal intent, including government oversight, profiling, personal attack, manipulation and legal litigation,” says the article. Government oversight and the mining of data by “big brother” is a particularly scary notion – and the legislation governing this is so confusing and willy-nilly that I’d say it’s a safe bet that if it’s not written down that a government agency can’t do something, it’ll be done.

What shocked me when reading this was the author’s statement that “cloud computing service providers like Google are exposed to legal and financial risks that could lead to their demise.” That’s a pretty extreme statement, and rather than get into whether I think that could happen or not, my advice to any company considering migrating to or expanding cloud usage is to focus more on prevention.

Instead of taking the “sitting duck” attitude, do everything proactively that you can to prevent service interruptions, hacking, security violations and other disasters happening to you.

So, what can you do? An end-to-end solution is what’s needed, in my view, including:

EXTERNAL END-TO-END MONITORING, including:

• Monitoring Frequency – from 1 minute to 60 minutes
• Multiple Check Locations – America, Europe, Asia and Australia
• Custom Monitoring Locations – possibility to setup your own monitoring locations
• No False Alarms – failures verified across multiple locations
• Monitors Websites, EMail Servers, Firewalls, VoIP, Databases, Domain Name Servers, Routers, Web Servers from end user perspective
• Supported Protocols – HTTP, HTTPS, FTP, SMTP, POP3, IMAP, SSH, PING, TCP, UDP, SIP, MySQL, DNS
• Web Page Content check – specify string to be checked for existence/non-existence on web page
• WebMap view – see all your servers and web sites in the single map view

SERVER AND NETWORK MONITORING, Providing Information on:

• Windows – CPU, RAM, Disk Usage, Processes, System Events, Installed Software
• Linux – CPU, RAM, Disk Usage, Processes, Load Averages
• FreeBSD – CPU, RAM, Disk Usage, Processes, Load Averages
• Solaris – CPU, RAM, Disk Usage, Processes, Load Averages
• SNMP Support – MIB browser, OID monitoring, SNMP Trap

APPLICATION (TRANSACTION) MONITORING, Offering:

• Monitis Transaction Recorder – record your web application flow
• Load time of each component of the page – check load time of each component of your web application
• Detailed troubleshooting – drill down multiple layers to find the root cause or component of the problem
• Multi-step support – multi-step application flow support
• Problem alerts specifying failed step – get immediate alerts specifying the step of your transaction where failure occurred
  

CLOUD STORAGE MONITORING Amazon S3:

• Monitor number of buckets – be aware when you reach some number of used buckets
• Monitor bucket size – find out disk utilization of your bucket or all the buckets in total
• Monitor objects in the bucket – find out how many objects exists in each bucket or in all of the buckets

CLOUD MONITORING of AMAZON EC2, RACKSPACE, Including:

• Monitor number of instances – be aware when you reach some number of EC2 running instances
• Monitor instance utilizations – CPU, RAM, Disk usage, Processes, System Events, Load Averages, Installed Software
• Policy specification – allow automatic deployment of internal agent on every new instance

GENERAL MONITORING Services:

• Instant Failure Alerts – E-mail, IM(Yahoo, GoogleTalk, ICQ), SMS, Twitter
  
• Schedule maintenance – define downtime periods during maintenance windows
• Escalation – escalate continuing problems to different staff members
• Alerting periods – specify alerting periods per contacts
• Performance Trends – graphical and table representation of performance and availabiliity over time(Current Day, Last 24 hours)
• History – option of digging into historical data
• SLA Reporting – detailed reporting with SLA metrics(Yearly/Monthly, Yearly/Weekly, Monthly/Weekly, Monthly/Daily
• Public reports – show your uptime to your customers
• Public widgets – show your uptime on your web site
• Weekly e-mail reports – get averages of the week by e-mail
• Daily e-mail reports – get averages of the day at the end of the day
• Monthly e-mail reports – get averages of the month in a monthly summary e-mails
• Time zone selection – adjust alert time zone
• Export results – export results in PDF/CSV in order to analyze them
• Timeout specification – specify timeout acceptable for your web site
• Failures From – specify monitoring locations affecting alerts(OR/AND condition is available)
• Continuous Alerting – get alerts until an issue is resolved
• Notify when back up – enable/Disable recovery alert
• Access Management – provide different level of access to your employees, customers
• Monitis API – fetch all the data directly without web access
• Monitis CLI – Monitis command line interface provides you library for fetching the data from your shell

In a piece I read on Networkworld, I was reminded just how much virtualization will become a successful component of cloud computing – and vice-versa.

Let me explain. There may be myriad definitions floating around out there about just what exactly is the cloud, but the goal of the cloud is very clear: to vastly improve the cost-effectiveness and dynamic provisioning of IT services. Better, faster, cheaper, says the story.

And to be even clearer about the value of the cloud, virtualization is included in the top considerations of what is most important in the new computing environment. Of course, virtualization is not a new concept (consider the existence of virtualized mainframes, VPNs and LANs). But in today’s computing world, almost every aspect of IT can be virtualized, including servers, desktops, applications, storage, network switches and routers.

The story also predicts that, more and more, we’ll see Input/Output on network switches, appliances such as WAN optimization controllers (WOC), application delivery controllers (ADC) and firewalls become virtualized.

While this all might seem pretty matter-of-fact to IT folks, the point here is that sometimes we have to take a step back and look at the broader view of how the cloud is changing almost every aspect of enterprise computing. Amazing!

Do you have a vision of IT staff disappearing as cloud computing develops and ushers in more automation and self-functioning tools? This implies that there won’t be anything to manage in a network.

In a recent InformationWeek blog that I came across, the author says that about three-quarters of the systems people she spoke with at a recent end-users conference were concerned about losing their jobs to the cloud.

I think this is a worst-case scenario (and a downright scary story) that IT folks are conjuring up. What will likely happen is that roles will change – not go away – as more expertise is needed in managing an enterprise’s eventual and measured migration to the cloud.

Systems administrators and networking professionals will be responsible for helping their organizations transition to a new IT environment, as well as for ensuring that there is a smooth migration path from an on-premise network to a hybrid environment (which would include on-premise, a private cloud and some components of a public cloud.)

To prepare for greater cloud computing management roles, IT folks can learn how to work more closely with other business units within their firms, and to help them understand new requirements, limits, and benefits of the new model. These “soft skills” have “often been lacking in IT organizations,” says the author. But soon enough, they’ll be critical – because cloud computing naturally involves the collaboration of several different business units. After all, it is a major change in the way a company computes.

So the reality really is that IT roles will grow from solely maintaining a network to more strategic roles in order to assist their organizations in managing the new cloud computing infrastructure.

IT folks can also prove themselves valuable in helping their organizations monitor service level agreements (SLAs), uptime statistics, data breaches and other cloud performance indicators. Taking on the watchdog role will bring new respect and appreciation from upper management.

There’s a fascinating article by Reuven Cohen
about cloud development in one of the fastest-growing and most interesting markets of the world: India. The country’s immense population (1.17 billion people) and growth of the middle class is spurring technological development. It’s like watching weeds sprout through pavement.

Cohen quotes Janakiram MSV (a fellow blogger and IT guy who works with Alcatel Lucent, Bell Labs-India) about why India is poised for significant cloud computing growth. He says that the Indian sub-continent is a “very unique and a potent geography” for cloud platform vendors due to the presence of an end-to-end IT ecosystem.

But beyond infrastructure, other trends are shaping cloud growth there, among them:

- Untapped market potential. Unlike the U.S. and Europe, India hasn’t hit saturation levels yet. So there’s opportunity at every level, such as enterprise or public sector.

- India is a test ground for piloting strategic adoption techniques, for example, engaging with ISVs in order to develop a platform or working with a mature developer community to pilot an SDK adoption plan.

- Small and mid-size Indian companies are eager customers for cloud vendors and services, due to the promise of low IT costs, data storage and a pay-as-you-use strategy.

Just to give you an example of the level of interest and development in cloud computing in India now, there’s a CloudCamp (an “un-conference” where early adapters of Cloud Computing technologies exchange ideas) India Tour going on right now that features five CloudCamp events throughout the end of February in some of the country’s largest cities.

I suspect we’ll be seeing a lot of cloud-related news coming out of India in the coming months and years, and it’s clear that the demand for cloud services should be skyrocketing, too.