Python development – scrap stream data

Looking for a Python scraping Developer in Boston MA area? Contact OT PROJECTS.


from lxml.html import fromstring
from lxml.etree import tostring

import urllib2


request=urllib2.urlopen("http://www.yahoo.com")



html=request.read()


try:

	root=fromstring(html)

except Exception, e:

	print "root", e
	continue
	
	
p_tags=root.xpath("//p[contains(@class,'stream-summary')]")




for p in  p_tags:
	try:
		print p.text
		print
	except Exception, e:
		print e
		exit()

print "End",



Looking for a Python scraping Developer in Boston MA area? Contact OT PROJECTS.

Python Developer – Send an email using Python

This example shows how to send an email (SMTP)
Full stack innovative Python Developer in Boston MA area Contact OT PROJECTS.
Make the needed changes where there’s *** in the code



import smtplib
import time
import sys

from email.mime.image import MIMEImage
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from email.header import Header
from email.utils import formataddr

print 'start'

#------------------------------------------------------
# change the parameters according to your SMTP settings
_my_name="*** Your name *** "
_smtp_domain='*** enter here your smtp server name ***'
_port=587 # put the right port
_login_user ='*** enter here the SMTP user name (usually your email) ****'
_password='*** enter here your password ***'
_from='%s <%s>' % (_my_name,_login_user)
_to='*** enter here the recipient email ***'
#------------------------------------------------------
html='

Hello,
This is an eample

' msg = MIMEText(html, 'html') msg['Subject'] = 'OT PROJECTS - a Python sample email procedure' msg['From'] = _from msg['To'] = _to msg['Reply-to']=_from tries=0 while tries<3: # for login try: email_server = smtplib.SMTP(_smtp_domain,_port) email_server.login(_login_user,_password) tries=4 # exit the try except: print 'email error, wait 10 sec.' time.sleep(4) tries+=1 # try again after 10 seconds if tries==3: sys.exit('Error login in to email server') try: rv=email_server.sendmail(_from, _to, msg.as_string()) except Exception as e: print 'Error sending email: ', e email_server.quit() print 'send email completed'

Python Developer in Boston MA area Contact OT PROJECTS.

Python Developer – Python socket example.

Socket server by default is TCP IP. Save the 2 codes in 2 files, run the server first.
* Note the client IP is pointed to 127.0.0.1, assuming the client and the server are on the same machine.

Python Backend Developer in Boston MA area Contact OT PROJECTS.

This is the server file code:



#!/usr/bin/python
import socket
import time

print "Socket server started"
socket_server = socket.socket()         
ip = '0.0.0.0' 
port = 5050 #choose any, larger than 1024 (otherwise you need to have admin/root permission to execute)               
socket_server.bind((ip, port))        

socket_server.listen(5)                 
while True:
   client, client_address = socket_server.accept()     
   print client_address, 'connected'
   client.send('Connection time: ' + str(time.asctime(time.localtime())))
   client.close()                


This is the client file code:



#!/usr/bin/python
import socket 
import time 
       
ip_of_the_socket_server = '127.0.0.1'
port_of_the_socket_server = 5050     # must be the server port number       

#send 10 messages, 10 sec delay between each
for i in range(1,11):
	my_connection = socket.socket()  
	my_connection.connect((ip_of_the_socket_server, port_of_the_socket_server))	
	print i,')', my_connection.recv(4096)
	my_connection.close
	time.sleep(10)




Python Developer in Boston MA area Contact OT PROJECTS.

Python Programmer – Python Decorators

A simple example to illustrate the use of function decorator in Python.

  • _func – you call a function that is executed.

With methods.



print 'started'		
def _exp(_func):

	def do_exp(v):
		s=0			
		for i in v:
			s+=(i)
		m_exp=s**2
		
		return {'exp':m_exp,'sum':s}
	return do_exp

	

@_exp
def show_exp():
	return

	

	
print show_exp([2,2,3])
print show_exp([1,2,3])
print show_exp([2,15,3,4,5,6,7,8,9,1,3])


With a class.


class _math(object):
	def __init__(self):
		print 'started'		
	def _exp(self,_func):
	
		def do_exp(v):
			s=0			
			for i in v:
				s+=(i)
			m_exp=s**2
			
			return {'exp':m_exp,'sum':s}
		return do_exp

	
m=_math()
@m._exp
def show_exp():
	return

	

	
print show_exp([2,2,3])
print show_exp([1,2,3])
print show_exp([2,15,3,4,5,6,7,8,9,1,3])

Full stack innovative Python Developer in Boston MA area Contact OT PROJECTS.

Python Developer – Python lists

Lists in Python are like Arrays, and are modifiable (in oppose to tuples).

Examples:

names=['Dan', 'Iris','Carla','Adam']
    The list length:
len(names)

Result:? 4

You can populate the list this way:


names=[]
names.append('Dan')
names.insert('Iris')
names+=['Carla','Adam']

    To print the names
?print names

Result: [‘Dan’, ‘Iris’, ‘Carla’, ‘Adam’]

    Print each name in names
for name in names:
	print name

Result:
Dan
Iris
Carla
Adam

You can do this as well:


for n in range(len(names)):
	print names[n]

Result:
Dan
Iris
Carla
Adam

Print the second value in names.
(Since sequence starts at index= 0, second is index is 1):


print names[1] 

Result: Iris

Print the values from the 2nd (index=1) to the 3th (you’ll need to assign index=3!)


print names[1:3]

Result: [‘Iris’, ‘Carla’]

Sort the list:


names= sorted(names)
print names

Result: [‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’]

Check if value exists:


print 'Dan' in names

result: True
note: Python is case sensitive.


print 'dan' in names

result: False

Modify the second value:


names[2]='Gary'
print names

Result: [‘Adam’, ‘Dan’, ‘Gary’, ‘Iris’]

Add a new value (between existing values) at index=2 (3rd location):


names[2:2]=['Karen']
print names

Result: [‘Adam’, ‘Carla’, ‘Karen’, ‘Dan’, ‘Iris’]


names.insert(2,'Karen')
print names

Result: [‘Adam’, ‘Carla’, ‘Karen’, ‘Karen’, ‘Dan’, ‘Iris’]

Delete a value:


names.remove('Karen')
print names

Result: [‘Adam’, ‘Carla’, ‘Karen’, ‘Dan’, ‘Iris’]
* note just one value was deleted.

Delete a value by index:


del names[2)
print names

Result: [‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’]

Count the appearances of a value:


print  names.count('Iris')


Result: 1

Working with multiple Lists:

instantiate


names=['Adam', 'Carla', 'Dan', 'Iris']
names1=names
names1+=['Deborah']
print names
print names1

Result:
[‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’, ‘Deborah’]
[‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’, ‘Deborah’]

Fork/copy


names=['Adam', 'Carla', 'Dan', 'Iris']
names1=list(names)
names1+=['Deborah']
print names
print names1

Result:
[‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’]
[‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’, ‘Deborah’]

another way by using the copy lib:


from copy import copy

names=['Adam', 'Carla', 'Dan', 'Iris']
names1=copy(names)
names1+=['Deborah']
print names
print names1

Result:
[‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’]
[‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’, ‘Deborah’]

Merge lists


names=['Adam', 'Carla', 'Dan', 'Iris']
names1=list(names)
names2=names+names1
print 'names:', names
print 'names1:',names1
print 'names2:',names2

Result:
names: [‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’]
names1: [‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’, ‘Gail’]
names2: [‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’, ‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’, ‘Gail’]

Full stack innovative Python Developer in Boston MA area Contact OT PROJECTS.

Python Programmer – Write and check JSON file with Python




import json
import pprint

#generate the JSON file
#will have 2 fields, one is an array (multiple records inside)

#*** make sure the fields and data are written in double quotes

my_json={}
my_json["description"]="This is a json file creation example written in Python"
my_json["data"]=[]
my_data_fields=['id','name']
for i in range(10):	
	my_json["data"].append({"id":i,"name":"name "+str(i)})

#take a look at the JSON output
print my_json

#write the file to the disk
fn='otp.json'
f=open(fn,"w")
f.write(json.dumps(my_json))
f.close()

#Now check if the JSON is valid (can be read as JSON)
json_data=open(fn)

data = json.load(json_data)
pprint.pprint(data)
json_data.close()

Python Developer- insert data records to a database (SQL)

Here is an example for inserting data to SQLite database.
You can see how to create a table, check if a table exists, and the use of single and multiple record insertion.


import sqlite3
import os



#-----------------------------------------------------------------------
#you need the path for the database
script_directory=os.getcwd()+os.path.sep

print script_directory

#this is the db we're creating

file_name="pydb.sqlite"
table_name='xx'
#Connect to the database
# Note - if an SQLite file desn't exists, the connect method will create it. If it exists, it'll open it.
con=sqlite3.connect(file_name)
#Open a cursor-this will move through a record set and read (or write)
cur=con.cursor()

#-----------------------------------------------------------------------

#Create a table, if not exist

sql=""" SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND tbl_name=? """
cur.execute(sql,[table_name])
table_exits=cur.fetchone()[0]
print table_exits
if table_exits==0:
	print 'create the %s table '% table_name
	# This code can run with any DB, not only SQLite
	sql=""" 
		CREATE TABLE %s
			(
				id INT,
				name VARCHAR(200)
			
			)
		""" % table_name
	#run the creation with the connect object.
	con.execute(sql)

#If you need a GUI for SQLite, one option is to use the FireFox add-on, works great.

#-----------------------------------------------------------------------
#-----------------------------------------------------------------------

#Insert statement for inserting new values
#it's alway good practice to spell the field names


sql="""INSERT INTO %s (id,name)""" % table_name


#The VALUES are 2 parameters that will follow later when executing
# the ? is a parameter

# Note - this may NOT work the same with different databases.

sql+=""" VALUES (?,?) """

#Here are 2 ways to add a record:
#-----------------------------------------------------------------------
#1) record by record 

#the parameters are a list

#inserting 2 records one by one
for i in range(1,3):
	
	params=[i,'Person '+str(i)] #'Person '+str(i) will make the names unique
	#execute via cursor 
	cur.execute(sql,params)
	# to save the new record, use commit method of the connection.
	con.commit()
	print 'run no. ',i
#-----------------------------------------------------------------------
#2) insert multiple (2) records - faster and efficient

#the parameters are lists in a list

params=[] # Note - you assign/declare the main list here
for i in range(3,5):
	params.append([i,'Person '+str(i)]) #append a list for each record
	
#execute via cursor 
cur.executemany(sql,params) # run all at once
# use commit method of the connection.
con.commit()
print '1 run  for all'
#-----------------------------------------------------------------------

Boston Django developer – Setting up SSL (nginx+Django)

Start with generating the SSL (here is a self-sign for development only):

Independent Django Developer in Boston MA area Contact OT PROJECTS.

Note: I use /home/user/ssl/ for the certification location – change accordingly




Topic:	SSL - django
Description:	create SSL #1
Sample:	openssl genrsa -des3 -out server.key 1024
------------------------------------------------------

Topic:	SSL - django
Description:	create SSL #1.1
Sample:	enter the pass: testssl
------------------------------------------------------

Topic:	SSL - django
Description:	Generate a CSR (Certificate Signing Request)  #2
Sample:	openssl req -new -key server.key -out server.csr
------------------------------------------------------

Topic:	SSL - django
Description:	Generate a CSR (Certificate Signing Request)  #2.1
Sample:	pass: testssl
------------------------------------------------------

Topic:	SSL - django
Description:	Generate a CSR (Certificate Signing Request)  #2.2
Sample:	Country: US
------------------------------------------------------

Topic:	SSL - django
Continue with writing accordingly:
Description:	Generate a CSR (Certificate Signing Request)  #2.3
Sample:	Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:massachusetts
Locality Name (eg, city) []:boston
Organization Name (eg, company) [Internet Widgits Pty Ltd]:OT PROJECTS
Organizational Unit Name (eg, section) []:development
Common Name (e.g. server FQDN or YOUR name) []:testssl
Email Address []:f@testssl.com

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:testssl
An optional company name []:testssl

------------------------------------------------------

Topic:	SSL
Description:	Remove Passphrase from Key #2.4
Sample:	        cp server.key server.key.org
                openssl rsa -in server.key.org -out server.key
------------------------------------------------------

Topic:	SSL
Description:	Generating a Self-Signed Certificate  #2.5
Sample:	        openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt
------------------------------------------------------

Topic:	SSL
Description:	Installing the Private Key and Certificate  #2.6
Sample:	        cp server.crt /home/user/ssl/conf/server.crt
                cp server.key /home/user/ssl/server.key
------------------------------------------------------



Now setup Nginx conf file:
if you use ubuntu: place a custom conf file here: /etc/nginx/conf.d
e.g.: /etc/nginx/conf.d/otp.conf:



server {
    listen          80;
    server_name     localhost;
    rewrite ^/(.*)  https://localhost/$1 permanent;
}

server {
    listen          4443 ssl;
    server_name     localhost;
    #rewrite ^/(.*)  https://localhost:4443/$1 permanent;
    access_log      /home/user/env/access.log combined;
    error_log       /home/user/env/error.log error;

    ssl_certificate         /home/user/ssl/server.crt;
    ssl_certificate_key     /home/user/ssl/server.key;

    location /static/ {
        alias /static/;
    }

    location /media/ {
        alias /media/;
    }

    location / {
        proxy_pass         http://localhost:8000/;
        proxy_redirect     off;

        proxy_set_header   Host              $http_host;
        proxy_set_header   X-Real-IP         $remote_addr;
        proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
    }

}






Independent Django Developer in Boston MA area Contact OT PROJECTS.

Restart nginx



 sudo /etc/init.d/nginx restart



Add to settings.py




SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True
SESSION_EXPIRE_AT_BROWSER_CLOSE=True




Now instead of localhost:8000 use localhost:4443

Finally, add this to your wsgi.py




os.environ['wsgi.url_scheme'] = 'https'



Should be the same, unless you used HTTP in your code/HTML or JS. Change accordingly.

Independent Django Developer in Boston MA area Contact OT PROJECTS.

Automate Testing a Web Application with Python

Automate testing a web application with Python using a wide range of powerful and relevant frameworks and tools. Splinter is one of the widely used testing tools with automated features that assess web apps using the popular programming language. It automates manual processes and tasks for testing web-based software apps, covering actions from opening the browser to checking its proper loading and app interactions.

Tools to Automate Testing a Web Application with Python

Splinter is a commendable automatic testing tool, which is an open source instrument that functions for testing multiple web apps using Python. Thus, Splinter’s testing tasks are written in Python and allow automated browser actions from visiting URLs to web interactions. The tool highlights built-in user-friendly functions that are designed for frequently used tasks.

Even a newbie will find it convenient and hassle-free to utilize Splinter and specific automated processes despite having limited Python scripting know-how. Selenium is another web app testing tool that offers usable abstraction layers for automated testing with Python.

Why Companies Prefer Python?

One of the relevant questions you need to answer when using the best programming language is why companies prefer Python. The backend language is one of the top-rated names in the industry at par with other popular programs such as Java, C, and C With its 25 years of existence, the programming language has its share of changes with the introduction of add-on features.

Python Then and Now

The first-ever Python 1.0 highlights the Modula-3 module system with Amoeba Operating System interaction while Python 2.0 is compatible with the Unicode Support and emphasizes the garbage collector feature with its introduction in 2000. Python 3.0 came to be in 2008 with a constructive design to prevent duplication.

Python for Modern Companies

Python 3.5 offers a wider range of additional features, making it popular among software development companies. The versatility of the programming language makes it quite popular along with its fewer programming codes. Almost 14% of programmers utilize Python with its compatibility to operating systems such as Mac OS, Windows, UNIX, and Linux.

Independent Python Developer in Boston MA area Contact OT PROJECTS.