Some thoughts after the election aftermath: final thoughts

So that is my view of the current situation. The impetus lies with the Conservatives and the SNP, and politics over the next five years may well be defined by the relationship between those two parties.

I expected the result of this election to be a mess, and even considered another election in November as being likely. I was wrong, and I take little condolence from the fact that nearly everyone else was wrong as well.

This is how I rate the position of the political parties:

• Conservatives are undoubtedly in the best position.
• SNP have done a stellar job, and are in a prime position for the negotiation about the future of the UK.
• UKIP have built a very firm base, but must be slightly disappointed they did not get more MPs.
• Greens have made solid progress, but again they deserved more MPs.
• Labour have lost an election they should have won, and also lost much of their intellectual heart.
• Liberal Democrats are suffering. There is a way back for them, but it will be a long process.

One last note of warning about the leadership elections that face Labour, the Liberal Democrats and possibly UKIP. An orderly changeover to a competent person can play wonders for a party. A disorderly, argumentative changeover to a competent person can irrecoverably harm the new leader. As an example, Ed Miliband never fully recovered from his somewhat disruptive election as leader. The candidates would be wise to remember that as they fight amongst themselves.

Another last note: it is odd to think that the leaders of the main UK parties - Conservative, Labour, Liberal Democrat and perhaps UKIP - might well all be different at the 2020 general election when compared to this one. I've a good feeling that that is (yes, you've guessed) unprecedented.

Some thoughts after the election aftermath: the Conservatives

Finally there are the Conservatives. What happened last week was unprecedented (yes, that word again), and it will be hard to repeat it in 2020, especially in the face of further unpopular austerity. But the are lucky in their enemies: Labour and the Lib Dems are in chaos, both in terms of leadership and strategic direction. UKIP are a threat, but they are currently riven by infighting and may not be able to keep facing both left and right for much longer.

A key issue for the Conservatives is that Cameron has admitted that he will not contest a third general election. This means that some time in the next five years there will be a new Conservative leader. I'm guessing that this changeover will occur after the 2017 EU referendum, whether he wins or loses. Therefore their success in the 2020 general election will depend on who is appointed as Cameron's successor. Boris Johnson is the media's favourite, and someone like Michael Gove is also likely to throw his hat into the ring. However my favourite is currently Rory Stewart, the MP for Penrith and the Border since 2010. He has a true boy's own backstory, including helping to run two Iraqi provinces. Even better, he is young and his constituency is in the north, a factor I will address below.

The party's positioning will be vital. Some of the elements that led them to be labelled 'the nasty party' have moved over to UKIP. Despite this, the Conservative vote share remained more or less static: those voters who moved to UKIP were offset by new Conservative voters from the Lib Dems and elsewhere. As such, Cameron's centrist approach has been vindicated. If the new Labour leader lurches to the left, as is entirely possible, then  the political centre ground may be left to the Conservatives. If Labour tries to be centrist, they will find the Conservatives already there.

There can be little doubt that the thin Conservative majority will decrease over this term through natural wastage: by-elections are always difficult for sitting governments to win. For this reason the Conservatives need to take an aggressive approach to 2020 and try to take more marginal seats. Experts were expecting them to take a defensive approach this year but they did the opposite, and actually attacked Labour marginals, of which they won enough to more than compensate for those they lost to Labour.

A key factor for the Conservatives will be the boundary changes they tried to pass in the last parliament, and which were somewhat controversially blocked by the Liberal Democrats. The changes were of two types: the equalisation of seat sizes in terms of populations, and the reduction in the number of MPs from 650 to 600. Combined, these moves would reduce Labour's built-in electoral advantage which means the Conservatives need to get a significantly greater vote to win the same number of seats as Labour. This is true if they keep the seat equalisation measures and remove the more controversial reduction in the number of seats. Expect this measure to be back in parliament this term.

If you look at the electoral map, it is clear the Conservatives powerbase is in the south and rural areas. Where they lack seats is in London, the urban north, and Scotland. London will prove a hard nut for them to crack, especially in the face of demographic changes and the mayoral elections which are almost certain to go to Labour. Scotland would also be difficult, but it would be sensible for them to target the seven second-places they achieved in Scotland this time. They might even gain one or two.

The real opportunity for the Conservatives lie in the urban north. It is telling that the third tweet George Osborne sent out after the election victory was about the 'northern powerhouse'. There can be little doubt that successive governments have treated the urban north poorly; the powerhouse of London has attracted more than its fair share of both government attention and finance.

There are many northern constituencies that might turn blue if the Conservatives gave them a little love; there can be little doubt that such attention would also be to the advantage of the country's finances as well: north versus south does not have to be a zero-sum gain. It would be easy to give the northern towns (and the north in general) some love whilst not spurning the south. The key to a third Conservative term in power may just lie in Osborne's 'northern powerhouse'. And he knows it.

Some thoughts after the election aftermath: the SNP

The SNP had a great election, but the result was not quite what they would have wanted. They are now a large minority party, whilst they would have been hoping to gain more power in Westminster by supporting (whether by coalition or confidence and supply) the Labour party.

The next big test facing the SNP is the 2016 Scottish elections, where they would be hoping to maintain the impetus gained in this general election. For various reasons, not least the electoral system in Scotland, this will prove more difficult than it did at this general election. After this there will be the 2017 EU referendum, and then the 2020 general election where they will be defending many seats that Labour, the Lib Dems and the Conservatives will be coveting.

There may also be an issue with the quality of their new MPs. Many would never have expected to be elected when they were selected to stand in what were safe seats for other parties, and many are very politically inexperienced. The SNP will have to carefully manage their new MPs to ensure that they do not do anything too naive.

They will also need to be careful not to be too strident in their opposition to the government. Support for independence has not particularly shifted since the independence referendum, which means that a proportion of the SNP's support comes from people who are against independence. This may seem odd, but given Scottish Labour's longstanding contempt for the Scottish electorate - as best seen by the Falkirk shenanigans of a couple of years ago - I can understand why even a unionist might vote for them. They will need to maintain this broad support if they want to keep many of their Scottish seats, some of which have small majorities. On the other hand, the Conservatives suffered a prolonged decline in Scotland over the decades and are left with one MP; they have not managed to recover. It is perfectly plausible that Scottish Labour will not recover either.

Even the SNP voters who are in favour of independence may not have that aim as their foremost objective: for many the NHS, education, jobs or wages may be the most important factor. If the SNP threaten those, people vaguely in favour of independence might swap their vote. No party should ever forget the bread-and-butter issues.

One thing I do not see as an issue is Salmond versus Sturgeon. Salmond stood down as party leader after he failed to win the independence referendum; since then Sturgeon has taken the party to a new level. Salmond won a seat in Westminster at this election, and some foresee problems between the two of them. I do not; both are brilliant political operators who have different approaches: Salmond will be in Westminster issues whilst Sturgeon leads the party in Scotland. It may be an unbeatable combination.

But the SNP's main aim is independence. They can choose to oppose the government and exploit the fractures in the union in the hope that more of the Scottish population will become pro-independence, or work with the government to move to a halfway house within the union: for instance Full Fiscal Autonomy (FFA), with English Votes for English Laws (EVEL) in return. There are hazards for them in either option.

The SNP are on a high. Whether they remain there depends on how they use their power over the next couple of years.

Some thoughts after the election aftermath: UKIP

UKIP had an excellent general election, although the aftermath has been distinctly interesting. Whilst they only got one MP and their leader failed to gain a seat (for the seventh time of asking, which must be some form of record), their vote increased manyfold, with some candidates very close to winning. They also gained many more councillors and now control their first council. But they have turned victory into defeat very rapidly.

Their first big issue is Farage: he immediately stood down as leader after the election, as he said he would if he failed to be elected. However he did not rule out standing again. Only a few days later came the controversial meeting of the party's leadership, where his resignation was declined. Reports vary about what went on in that meeting, but whatever the truth it led to open warfare within the party. The main combatants are Farage himself and the party's only MP, Douglas Carswell.

The perceived wisdom was that UKIP would hurt the Conservatives; however at the election they took more votes from Labour. But UKIP cannot maintain this position for long, and the new leader will need to work out whether they are going to face to the left or right in the future.

UKIP seem to have gained the Liberal Democrats' 'None of the above' position as a protest party. Whilst many UKIP supporters are very supportive (as web forums show), they undoubtedly gained many votes from people who disliked the other main parties. This was exactly the same position that the Liberal Democrats formerly held, albeit the individual voters would mostly be different. Such voters could easily shift back again. Therefore UKIP will have to continue to position themselves in voters' minds as being 'different' to the other parties. That position may be hard to maintain if they become more successful, or the if the infighting continues for too long.

A 2017 referendum on EU membership may shoot one of UKIP's most precious foxes. However it may also play straight into their hands. It would be good if the party could decide on how a UK outside the EU would work, but that currently seems beyond them. Will they support or oppose the referendum? The hatred and distrust that many members show for David Cameron shows that they may well oppose anything he proposes.

I am unsure what UKIP should do with their immigration calls. On one hand mass immigration has caused social problems; on the other, UKIP have problems with racists within their ranks. Farage (or whoever leads the party) will have to try to clear out the more extreme elements and work out a position ready for the 2017 EU referendum.

They are largely a protest party, and there has been plenty to protest about. In particular the sexual abuse of children by gangs of men in Rotherham, Rochdale, Oxford and elsewhere has understandably got many people angry. These scandals have been widely ignored, especially by Labour. But they need to be careful not to concentrate on cases where the abusers are ethnic minorities and the victims white, as opposed to the many cases where the abusers are white and/or the victims ethnic. Sexual abuse should not be made solely into a racial issue.

In addition, UKIP have rallied against what they term as 'LibLabCon' and 'the establishment' - the latter of which Farage mentioned in his non-concession speech. Leaving aside the fact that Dulwich-educated ex-city trader and MEP Farage is as 'establishment' as it gets, the more power they have, the less effective that call will become.

However great opportunities for UKIP stand in the large swathes of constituencies where they came second. It is easy to see the party throwing resources into these areas. However there is the caveat that what people on the east coast want may be anathema to their urban supporters.

The wild card is Douglas Carswell. The Essex MP had a reputation as a deep thinker before he moved from the Conservatives to UKIP last year, and the fact that his move was not criticised more says a great deal for the way he is regarded (this contrasts with the other Conservative MP who made the same move, Mark Reckless, who was roundly pilloried and lost his seat last week). Carswell is an unashamed right-winger who must be concerned by any move of UKIP to the left, and as UKIP's sole MP he will have a great deal of power and influence over the party's direction.

I see it as unlikely that Carswell would stand for leader; instead I think he will be content as acting as the power behind the throne, particularly if his wisdom will affect the views of his many friends that remain within the Conservative party. But the events of the last week makes it possible that Farage will not remain in UKIP for ling: the party seems to small for both him and Farage.

UKIP need to be careful that they do not turn their victory at the Euro and General elections into a massive defeat. But that might mean they need someone other than Farage at the helm.

Some thoughts after the election aftermath: the Greens

The Greens had a mixed election. They stood in many more seats than ever before, but did not manage to increase their number of MPs, with Caroline Lucas remaining their sole representative at Westminster. The Green party leader, Natalie Bennett, failed to get elected in her seat of Holborn and St Pancras.

But they did well in the popular vote, getting over 1.1 million votes and 3.8% of the total, an increase of 2.8% over 2010. Like UKIP, the fact they only got one seat is down to the vagaries of the FPTP electoral system.

The council elections were less positive. The Greens ran Brighton and Hove council as a minority administration, but this year they lost seats and control over the council. In England they gained a paltry 9 councillors (compared to the Conservatives 497).

The question is what proportion of their new support comes from people who are now genuine Green supporters, and what proportion supported them because they found they could not vote for Labour or the Liberal Democrats. The former are more likely to stick with the party than the latter.

The EU referendum in 2015 has to be an opportunity for the Greens. Much depends on Cameron's renegotiation with the EU; but there is a place for an unashamedly pro-EU party. Ordinarily that would be filled by the Liberal Democrats, but they are currently a party in decline. It would be easy for the Greens to accept that torch and run with it.

Some thoughts after the election aftermath: the Liberal Democrats

The Liberal Democrats find themselves in a hideous position. In my view they did the right thing going into coalition with the Conservatives in 2010, and they generally behaved well whilst in power. Their record in government is one they should be proud of. But it came at a hideous cost. It turns out that much of their vote came from the 'none of the above' voter, and going into government destroyed that sector. The red liberals also shifted in numbers to the Labour party, although in nowhere near the numbers the latter were expecting, As a result the Liberal Democrat base, both in terms of members, councillors and MPs, is tiny.

So what can they do? The answer is not much before 2020. They need to elect a leader who can find them a new position in politics, and for that reason if I was them, I would delay choosing a new leader until after Labour. But whoever gets the job has a hard task ahead of them.

They need someone who can engage with the media and sell whatever policy position and vision they develop. Sadly, none of their eight remaining MPs really seems to fit that bill if you exclude Clegg.

They also need to rebuild their political base. The last five years have been rough for them, both in terms of MPs and councillors. Yet they still have a large base of committed activists who can help them rebuild. A promising sign is that membership numbers have been increasing in the days since the election.

There is not much else that can be said about the Liberal Democrats. That is part of their problem: with the rise of UKIP and the SNP they are approaching irrelevance. They need a leader who can reverse that trend, even if full recovery takes several elections.

Some thoughts after the election aftermath: Labour

Any Labour supporter must be really hurting after what was a terrible general election. So what happens now?

Labour need to analyse what went wrong and try to elect a leader who can correct those faults, a process they singularly failed to do after their defeat in 2010. Sadly, the immediate aftermath of the election has seen many Labour supporters, including some past and current politicians, refuse to face up to the problems that  the party faces.

Choosing a new party leader will be made more difficult by the pull from the left and the more centrist segments of the party. If they are not careful it will become 'Brownites versus Blairites part three', although long after the major players have departed the scene. A new leader might be best looking for another route - a fourth way after Blair's third way. Some have even started calling it 'Now Labour'.

I have no particular favourite candidate for the leadership, and it is still early days. Initially the media seemed to be concentrating on ex-soldier Dan Jarvis, whose backstory is certainly compelling. However backstory does not of itself make a good party leader, and I have never heard him make a speech to judge his ability. He has also come out to say that he will not stand for personal reasons.

The timing of the leadership election is an issue. In 2010 Labour waited until the autumn conference to choose their leader; this gave the coalition months in which they could pull themselves together and set the agenda. For this reason many are calling for a snap election. This would favour the old guard, people such as Andy Burnham who stood in 2010. A later election would favour someone less well known within the party. It currently looks as though the latter argument has won, with a leadership election possibly being held in mid-September.

Ed Miliband was only elected because of the union vote (including one union who sent out ballot papers to members along with literature favouring him; a strange thing to do in a democratic election). The leadership selection process was altered last year after the Collins Review to a One Man, One Vote system, This will reduce the unions' influence over the election. However the new leader will still have to maintain good relations with the unions who donate massive sums to the party.

Once elected, the next Labour leader will have to decide what to do about Scotland. Their unprecedented (yes, that word again) rout of Scottish Labour has helped the Conservatives to a limited majority. But some of the SNP majorities over Labour are small, and Labour will be eyeing those constituencies covetously come 2020. Their problem is that Scotland is more left-wing than England. If the new leader moves Labour to the left then he risks alienating English voters; if he moves to the right he does the same for Scottish voters.

That is not to say the correct positioning is impossible: someone with the political skills of Blair could satisfy these seemingly different electorates (even if 'Now Labour' seems rather hackneyed). The problem is that I cannot see Blair Mk II anywhere on Labour's benches. However Blair was not exactly a household name before he was made Labour leader after John Smith's death in 1994, and yet he changed Labour forever.

The new leader needs to address the main reasons they lost: strategy. Miliband put in place a '35% strategy', which meant gaining 35% of the vote. Strangely, this would have been barely enough to win a majority; a seemingly modest ambition. As it happens, they failed in even that. In my view, the problem was that Labour espoused no true vision.

Most of Labour's campaign was worthy, but the disparate points were only connected by a vague hatred of the Conservatives. Whilst that was enough to fire up the Labour core vote, it did nothing to attract the general population. Sadly for Miliband, the core vote of both main parties is only about 30%. The new Labour leader needs to develop a vision for the country that embraces people outside their core vote. Blair managed this; Foot, Kinnock, Brown and Miliband did not.

There is also the need to sort out, and preferably widen, the party's finances. Currently they are too reliant on income from the unions, who are all too willing to use the influence their cash gives them. That union influence can have positive outcomes; however at times, such as when the unions essentially elected Miliband to the leadership, it can be positively malign.

Labour need an 8% swing to get an overall majority. Add in the boundary changes that the Conservatives are keen to push through, and it may be nearer 12%. This will be a difficult task: in 2010 Cameron got a 5% swing, whilst Blair got 10.3% in 1997. So the new leader may well need a shift larger than Blair achieved. For this reason, Labour's recovery may be a two-term task, and they may want to select a leader who can see them through that process.

The near-wipeout of Scottish Labour has also has ripped out much of the intellectual heart of the party. From Keir Hardie to Gordon Brown, much of Labour's political philosophy has come from Scotland. In this election they lost a large number of the party's deepest thinkers. That may hurt them more than is currently obvious.

Labour faces deep problems on all sides. Hopefully they can find a leader that will be able to solve them.

Some thoughts after the election aftermath: polling

There can be no defence: the opinion polls got the 2015 general election disastrously wrong.

In my view this error was not due to a late swing to the Conservatives as many are saying; in fact, during the last week of the campaign some of the phone pollsters converged with the online polls to show Labour and the Conservatives on level pegging. This is embarrassing for those phone pollsters as it turned out that they had been more accurate than the online pollsters.

The odds are that the polls, and especially the online pollsters, had been underestimating the Conservatives and overestimating Labour for many months, perhaps years. If this is the case, then any pollsters who try to 'fix' the problem of late swings will miss the real problem.

Another problem was the sheer number of polls. Yougov did at least five online polls a week in the run-up to the election. This data swamped the other pollster's efforts and thoroughly skewed the narrative. As online polls performed the worst, the inaccurate Yougov polls incorrectly influenced the political mood.

Labour are even saying that it may have altered the result of the election; if broadcasters had known the Conservatives were ahead, they would have been given a harder time. Whilst this seems a rather self-serving, silly argument, there may be some truth in it.

The Conservative's adviser Jim Messina has cast scorn on the public pollsters.

"I think most public polling is garbage and is wrong," he said. "Almost every public poll had this race tied the night before the election. We had us winning 315 seats … I think that most public polling should be shot. It's ridiculous."

From the pollster's dire performance, he is right. His own data seemed much more accurate. It was apparently gathered from varying sources, including online. That may well turn out to be the future for polling.

The Labour party have also said that their internal polling showed a very different picture to that of the public pollsters, and again this puts the accuracy of the public polling further in doubt.

It is easy to see several pollsters pulling out of UK political polling, or even going out of business altogether.

If they stay in, they need to ask themselves the following questions:

1. Why were the final polls so inaccurate?
2. Why did they all seem to converge on the same, incorrect, answer? We it just down to methodological changes, or were there deeper structural problems in the way polling is performed? 

They also need to stop tinkering with their methodologies. Altering weightings and other factors a few weeks before a major election makes comparisons with other polls impossible. Such changes also tend to be hidden from the casual layperson reading the polls, meaning that shifts caused by methodological changes appear to be changes in support for the parties.

But that is in the past. What are the challenges facing the parties?

Some thoughts after the election aftermath: prologue

Well, that was unexpected. The 2015 general election turned out to have a more surprising result than even the famed 1992 election, where the Conservatives also won a surprise victory over Labour. I, like many other people, believed the opinion polls even as I wondered why the twin effects of Miliband's low popularity and a recovering economy were not helping the Conservatives.

The mistake was probably understandable. The odds were against the Conservatives winning a majority: no party in power has actually increased its number of seats for nearly a century, so I had more-or-less discounted that option. That was a mistake on my part: unprecedented can easily become precedented.

Also the mood music in the media was against it. Nearly everyone seemed to agree that a coalition would be the end-result; nearly everyone got it very wrong. Group-think prevented too many people from seeing the reality.

So why did the Conservatives win?

There was a difference between the way the two main parties canvassed. Labour went in for a traditional scheme where people canvassed locally, with ministers and celebrities performing high-profile visits to marginals. The Conservatives did some of this, but also had what they called 'Team 2015'; a team of over 40,000 young, enthusiastic supporters who were bussed into target constituencies to canvas. This approach was trialled and honed at the Newark by-election in the last parliament, and seems to have worked wonders. Labour got TV time as Eddie Izzard pranced around the country, whilst the Conservatives got votes.

If 2015 proves anything, it is that celebrities do not sway votes. That must hurt some celebrities who seem to have rather over-developed sense of their own importance.

The 2015 election was also a massive success for the leadership team David Cameron assembled. Both main parties hired advisers who had helped Obama to power. Whilst Labour had David Alxerod who managed to spell Miliband's name incorrectly, the Conservatives had Axelrod's friend, Jim Messina, who seemed to work well with the Conservative's campaign chief Lynton Crosby.

The differing approaches of the two campaigns can be seen online. A glance at Facebook or Twitter in early May would have led you to think that Labour were going to get 125% of the vote. My timeline was half-filled with Labour supporters throwing disdain on the Conservatives, whereas the only posts I received from the Conservatives were from the official campaign.

That was part of the problem: some of the Labour supporters were so strident, so nasty, that it was actually a turn-off. Meanwhile, the Conservatives used Facebook and Twitter in the way customers are supposed to use them: allegedly they spent £100,000 a month on Facebook alone. That did not just go on adverts to push in front of people, it also went on gathering demographic data on individuals.

This data allowed them to target the seats they needed to get a majority whilst Labour supporters talked (and sometimes shouted) amongst themselves. This work allowed Messina and Crosby to confidently state a week before the election that the Conservatives would win, whilst Miliband was polishing his victory speech a few minutes before the devastating election poll was released.

To a large extent it was Facebook and Twitter wot won it, although not in the way Labour supporters expected.

So what about the polling?