分類
程序

在 VS Code 中調試 Django Q 任務

在 .vscode 文件夾中創建一個 lanch.json 文件, 內容如下:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Django Q",
            "type": "python",
            "request": "launch",
            "program": "${workspaceFolder}/manage.py",
            "args": [
                "qcluster"
            ],
            "django": true,
            "env": {
                "DJANGO_SETTINGS_MODULE": "YOUR_PROJECT_NAME.settings"
            },
            "gevent": true
        },
        // Your other configurations go here
    ]
}

只需修改兩個地方:program 裏的 ${workspaceFolder} 是變量,如果你項目的 manage.py 在 VS Code 項目根目錄下,那這行就不用修改。 DJANGO_SETTINGS_MODULE 這裏需要把 YOUR_PROJECT_NAME 替換成你項目的名字。然後按 Debug 按鈕就可以調試 Django Q task 了。

對於這個問題,ChatGPT 4 前前後後給出了五次錯誤配置,不得不記錄一下。

分類
程序

使用 Cloudflare Worker 中轉 HTTP 請求

Cloudflare Worker 可以方便的中轉 HTTP 請求,下面示例是我之前用過的,算是密碼保護的中轉特定請求。其中的 X_Custom_PSK 算是密碼,在 Settings > Variables 設置,這樣就只有我的程序可以請求。

addEventListener("fetch", event => {

  const psk = event.request.headers.get("X_Custom_PSK");
  if (psk === X_Custom_PSK) {
    event.respondWith(handleRequest(event.request));
  }else{
    const failed_response = new Response('Sorry, you have supplied an invalid key.', {
      status: 403,
    });
    event.respondWith(failed_response);
  }
})

async function handleRequest(request) {
  const resp = await fetch('https://domain.ltd/checkpoint/list', {
      method: 'POST',
      headers: {
          'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.3 Safari/537.36',
          'Accept': '*/*',
          'Accept-Language': 'zh-CN;q=0.8,zh;q=0.6,en-US;q=0.4,en;q=0.2',
          'Origin': 'https://domain.ltd',
          'DNT': '1',
          'Connection': 'keep-alive',
          'Referer': 'https://domain.ltd/',
          'Sec-Fetch-Dest': 'empty',
          'Sec-Fetch-Mode': 'cors',
          'Sec-Fetch-Site': 'cross-site'
      },
      body: new URLSearchParams({
          'page': '1',
          'pageSize': '10',
          'latitude': '22.5',
          'longitude': '114.0',
          'queryType': '0'
      })
  });
  return resp;
}

下面這個則是用一個 worker 代理多個網站。

addEventListener("fetch", event => {
  let url=new URL(event.request.url);
  if (event.request.url.indexOf('shaman')>-1){
      url.hostname="ft.shaman.eu.org";
  }else{
      url.hostname="www.rfi.fr";
  }  
  let request=new Request(url,event.request);
  event.respondWith(fetch(request));
});
分類
Linux 程序

Django簡易搭建上傳文件

這裡使用Django+Gunicorn+Nginx的方式簡單運行一個小型webserver,實現一個簡單的上傳文件到服務器的功能(並不生成下載鏈接)。

啟動虛擬環境,安裝django和gunicorn:

pip install Django==2.0
pip install gunicorn
#進入要放置代碼的目錄並新建項目
django-admin startproject mysite
##或者在當前目錄建立項目
#django-admin startproject mysite .
cd mysite
#新建app
python manage.py startapp polls

先建立一個表格:

#polls/forms.py
from django import forms

class UploadFileForm(forms.Form):
    title = forms.CharField(label='密碼',max_length=20,widget=forms.PasswordInput)
    file = forms.FileField(label='文件',)

修改view:

#polls/views.py
import os
import subprocess
from django.core.files.storage import FileSystemStorage
from django.conf import settings
from django.shortcuts import render
from django.http import HttpResponse
from .forms import UploadFileForm

#handle file example with file
def handle_uploaded_file(f):
    with open('/file/should/be/saved/here/target.odt', 'wb+') as destination:
        for chunk in f.chunks():
            destination.write(chunk)

#another handle file example with filename
def handle_uploaded_file2(filename):
    msg=''
    try:
        targetZipFilePath = os.path.join(settings.BASE_DIR, filename)
        cmd1=subprocess.check_call(["unzip", "-o", targetZipFilePath, "-d", "/home/fred/workspace/"])
        if cmd1==0 :
            cmd2=subprocess.check_call(["cp", "-Rf", "/home/fred/workspace/dist", "/home/fred/"])
            if cmd2==0 :
                msg="deployed successfully"
            else:
                msg="error 2"
        else:
            msg="error 1"
    except:
        msg = 'error 0'
    return msg

def upload_file(request):
    if request.method == 'POST':
        form = UploadFileForm(request.POST, request.FILES)
        if form.is_valid() and request.POST['title']=='Secret':
#            handle_uploaded_file(request.FILES['file'])
#            return HttpResponse("上傳成功")
            myfile = request.FILES['file']
            fs = FileSystemStorage()
            filename = fs.save(myfile.name, myfile)
            uploaded_file_url = fs.url(filename)
            print(uploaded_file_url)
            res = handle_uploaded_file2(uploaded_file_url)
            return HttpResponse(res)
    else:
        form = UploadFileForm()
    return render(request, 'upload.html', {'form': form})

新建一個表格的模板:

#polls/templates/upload.html
<form enctype="multipart/form-data" action="/polls/upload/" method="post">
    {% csrf_token %}
    {{ form }}
    <input type="submit" value="上傳" />
</form>

新建一個url路由表:

#polls/urls.py
from django.urls import path

from . import views

urlpatterns = [
    path('upload/', views.upload_file, name='upload_file'),
]

修改項目路由:

#mysite/urls.py
from django.contrib import admin
from django.urls import include,path

urlpatterns = [
    path('polls/', include('polls.urls')),
    path('admin/', admin.site.urls),
]

修改項目設置:

#mysite/settings.py
INSTALLED_APPS = [    
    'polls.apps.PollsConfig',
#    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
#    'django.contrib.messages',
#    'django.contrib.staticfiles',
]
LANGUAGE_CODE = 'zh-Hant'
TIME_ZONE = 'Asia/Taipei'

然後在項目目錄(最上層)運行gunicorn就可以訪問了:

gunicorn mysite.wsgi --bind 127.0.0.1:3040

nginx中增加如下server即可在外網訪問了(鏈接應該是http://YourPublicIP:8081/polls/upload/):

server {
        listen 8081;
        server_name 127.0.0.1;
        charset utf-8;
        keepalive_timeout 60s;
        #access_log logs/django2a.access.log combined if=$loggable;
        
        location / {
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_redirect off;
    
            if (!-f $request_filename) {
                proxy_pass http://127.0.0.1:3040;
                break;
            }
    }
}

CentOS6+Django2+MySql

yum install MySQL-python
pip install mysqlclient
#如果import MySQLdb提示無libmysqlclient.so.18
#則建立軟連接如下
ln -s /usr/local/mysql/lib/libmysqlclient.so.18 /usr/lib64/libmysqlclient.so.18
分類
程序

把媒體中地理位置標籤從 WGS-84 轉換成 GCJ-02

手機拍攝的照片地理位置默認是 WGS-84 系統,但是谷歌地圖使用的卻是 GCJ-02 系統,這就會出現偏移。爲了展示正確的位置,可以通過把照片的地理位置標籤轉換成 GCJ-02 來實現。需要用到 Phil Harvey開發的 ExifTool 和 Avatar sshuair 開發的 coord-convert。前者下載exiftool.exe 到本地即可,後者可以通過 pip 安裝。

# -*- coding: utf-8 -*-

#pip install coord-convert

#Usage:
#Drage a single photo or directory with photos to this script
# OR
#python wgs2gcj.py D:\my.JPG
#Results will be wrote to gps.log at the same dir with this script

from coord_convert.transform import wgs2gcj
import subprocess,json,sys,os,time,datetime
#You may set your exiftool file here
exiftool_path = "D:\\Program Files\\portable\\exiftool.exe"

work_dir = os.getcwd()
def log42(logFile,logText):
	ts = int(time.time())
	dt = datetime.datetime.fromtimestamp(ts).strftime('%Y-%m-%d %H:%M:%S')
	
	log_file = os.path.join(work_dir,logFile+".log")
	with open(log_file, "a") as myfile:
		myfile.write(dt+" "+logText+"\n")
		print(logText)
		
		
if (exiftool_path == ""):
	exiftool_path = os.path.join(work_dir,'exiftool.exe')
if not os.path.exists(exiftool_path):
	log42("gps","Error: exiftool.exe not found neither in script directory nor in the script setting \"exiftool_path\"")
	exit()
	
file_paths_input = sys.argv[1:]
file_paths = []

for file_path_input in file_paths_input:
	if os.path.isfile(file_path_input):
		file_paths.append(file_path_input)
	elif os.path.isdir(file_path_input):
		for root,ds,fs in os.walk(file_path_input):
			for ff in fs:
				full_path = os.path.join(root,ff)
				file_paths.append(full_path)

for f in file_paths:
	output = subprocess.check_output('"'+exiftool_path+'" -struct -j -a -n "-gps*" '+f, shell=True)
	data = json.loads(output)
	if 'GPSMapDatum' in data[0]:
		log42("gps",data[0]['SourceFile']+':'+str(data[0]['GPSMapDatum'])+' '+str(data[0]['GPSLatitude'])+' '+str(data[0]['GPSLongitude']))
		if(data[0]['GPSMapDatum']=='WGS-84'):
			gcj_lon, gcj_lat = wgs2gcj(data[0]['GPSLongitude'], data[0]['GPSLatitude'])
			log42("gps",str(gcj_lat)+' '+str(gcj_lon))
			if (gcj_lat==data[0]['GPSLatitude']):
				log42("gps",'not in China')
			else:
				output = subprocess.check_output('"'+exiftool_path+'" -overwrite_original -exif:gpsmapdatum=GCJ-02 -exif:gpslatitude='+str(gcj_lat)+' -exif:gpslongitude='+str(gcj_lon)+' '+f, shell=True)
				log42("gps",output.decode())
		else:			
			log42("gps","not a wgs84 location")
	else:
		log42("gps","no gps data found")
		
log42("gps","task finished")

另請參考:使用ExifTool處理文件元數據wgs-84-gcj-02-經緯度在線轉換

分類
程序

PHP 使用 Nominatim 的逆地理編碼

之前一直使用谷歌的逆地理編碼服務,多年來運行良好。但是谷歌畢竟是一家巨大的商業公司,所以我現在改用 Nominatim

Nominatim(來自拉丁語,“按名稱”)是一個通過名稱和地址搜索開放街圖數據,根據開放街圖點來生成合成地址(逆地理編碼)的工具。
<?php
header("Access-Control-Allow-Origin: *");
header('Content-type: text/json');
header("Cache-Control: no-cache, must-revalidate");

$lat=$_POST["lat"];
$lng=$_POST["lng"];
$lang=$_POST["lang"];

$url = "https://nominatim.openstreetmap.org/reverse?format=geojson&lat=".$lat."&lon=".$lng."&accept-language=".$lang;

$ch = curl_init(); 
//set your own agent name
$agent = 'location/0.2(ft.shaman.eu.org)';
curl_setopt ($ch, CURLOPT_URL, $url); 
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1); 
curl_setopt ($ch, CURLOPT_CONNECTTIMEOUT,20); 
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
$content = curl_exec($ch); 
curl_close($ch);

$content = json_decode($content);
$resultArray = array();
foreach($content->features[0]->properties->address as $key => $value) {
    //remove country_code and postcode
    if ($key=='country_code' or $key=='postcode'){            
    }else{
        array_push($resultArray,$value);
    }
}
echo implode(",",$resultArray);
?>	
curl -d 'lat=22.57776&lng=113.94849&lang=zh-TW' https://your.api.domain/api.php
西丽华昌大厦,西丽南路,松坪村,南山区,西丽街道,广东省,中国
分類
程序

使用python同步本地時間

服務器超售到不行,cpu時間隔一段時間就滿好多。嘗試配置了標準了自動的時間同步,效果不理想。於是用python調用worldtimeapi.org的api配合crontab來修正系統時間。

import subprocess,requests,time

def main():
    try:
        response = requests.request("GET", "http://worldtimeapi.org/api/timezone/Asia/Hong_Kong")
        timeJson=response.json()
        timeAbs = abs(timeJson['unixtime']-time.time())
        if timeAbs > 15 :
            subprocess.call(['date','+%T','-s',timeJson['datetime'][11:19]])
            print("time synced")
        else:
            print(timeAbs)
            
    except:
        print("net error")
    
if __name__ == '__main__':
    main()
#添加crontab任務
crontab -e
#每兩小時檢查一次
0 */2 * * * /usr/local/bin/python3 /home/42/time.py >> /home/42/t.log 2>&1
分類
程序

python使用pyftpdlib實現ftp服務

需要使用ftp分享文件,記得很久之前折騰過ftp服務的搭建,賬號權限設置都挺複雜,這次使用python和pyftpdlib實現,幾分鐘就搞定啦。

#安裝pyftpdlib,當然還是推薦在虛擬環境裡裝
pip install pyftpdlib

然後新建一個python文件myFtp.py,內容如下:

import os

from pyftpdlib.authorizers import DummyAuthorizer
from pyftpdlib.handlers import FTPHandler
from pyftpdlib.servers import FTPServer

def main():
    # Instantiate a dummy authorizer for managing 'virtual' users
    authorizer = DummyAuthorizer()

    # 第一行新建一個具有讀寫權限的用戶(用戶名,密碼,ftp文件夾)
    # 第二行新建匿名讀取賬戶
    authorizer.add_user('user', '12345', '.', perm='elradfmwMT')
    authorizer.add_anonymous(os.getcwd())

    # Instantiate FTP handler class
    handler = FTPHandler
    handler.authorizer = authorizer

    # 設置一個個性化歡迎語 (客戶端來連接的時候會返回給它)
    handler.banner = "基於pyftpdlib的ftp服務已準備好"

    # Specify a masquerade address and the range of ports to use for
    # passive connections.  Decomment in case you're behind a NAT.
    #handler.masquerade_address = '151.25.42.11'
    #handler.passive_ports = range(60000, 65535)

    # 監聽任意ip目標的2121端口,ftp標準端口是21
    address = ('', 2121)
    server = FTPServer(address, handler)

    # set a limit for connections
    server.max_cons = 256
    server.max_cons_per_ip = 5

    # start ftp server
    server.serve_forever()

if __name__ == '__main__':
    main()
#最後運行就可以了
python myFtp.py

其實這次也是走了彎路的,那就是先嘗試了Twisted,stackoverflow的網友說可以一行命令實現ftp服務。結果pip裝不上,需要下載了安裝包用

pip install 
裝。裝好後匿名用戶可以跑,設置了密碼就顯示密碼錯誤,搜索一番也沒找到辦法,於是使用了pyftpdlib。雖然實現ftp服務失敗了,但Twisted確實是一個很厲害的項目,有時間的話可以學習一下。 Twisted also supports many common network protocols, including SMTP, POP3, IMAP, SSHv2, and DNS.