分類
程序

免費使用 AWS Lambda 的一個示例

這裏演示一下免費使用 AWS Lambda 創建一個查詢網頁大小的接口。之所以直接使用 Lambda 生成的鏈接而不是 API Gateway 的鏈接,是因爲 API Gateway 的免費額度是在註冊帳號的一年內有效。而且 API Gateway 最大超時只有 30 秒,而 Lambda 可以設置到 15 分鐘。

首先註冊一個帳號,我記得只要郵箱和手機號就行,不用綁定支付方式。然後在控制台搜索 Lambda 進入 Lambda 控制台。點擊 Funtction > Create function. 選擇 Author from scratch,輸入函數名字沒有特殊要求,Runtime 選 Python 3.10,Architecture 選 x86 和 arm64 都可以,各有免費額度。點開 Advanced settings,勾選 Enable function URL,勾選 None。CORS 看需要也可以勾上,然後點擊右下角 Create function 就可以了。

在函數頁面 Code 標籤頁貼上如下代碼:

import json
import os
import random
import urllib.request

UA = '''Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edge/19041.423
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 OPR/87.0.4390.99
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Vivaldi/6.1.3035.111
Mozilla/5.0 (Windows NT 10.0; WOW64; rv:85.0) Gecko/20100101 Firefox/85.0
Mozilla/5.0 (Windows NT 10.0; Win64; x64; Trident/7.0; rv:11.0) like Gecko
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.51 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36'''

US_LIST=UA.split()

def lambda_handler(event, context):
    event_body_str = event['body']
    event_body = json.loads(event_body_str)
    # print(str(event_body))
    if event_body['secret'] == os.environ.get('secret'):
        try:
            t_ua = random.choice(US_LIST)
            res = urllib.request.urlopen(urllib.request.Request(
                url=event_body['url'],
                headers={"User-Agent": t_ua,},
                method='GET'),
                timeout=180)
            content = res.read()
            return {
                'statusCode': 200,
                'body': json.dumps(str(len(content))+' '+event_body['url'])
            }
        except:
            return {
                'statusCode': 500,
                'body': json.dumps('request failed.')
            }
    else:
        return {
            'statusCode': 500,
            'body': json.dumps('unauthorized.')
        }

secret 的環境在 Configuration > Environment variables 中添加。在 General configuration 中可以設置函數的超時時長 Timeout,默認是 3 秒。在 Function URL 裏可以看到我們需要的 URL 類似:https://4xyqiuexi7l3fcnuyuevjgclei0jhpuy.lambda-url.us-east-1.on.aws/ 。使用如下 curl 命令測試下:

curl -X POST -H "Content-Type: application/json" -d '{"secret":"Dn2Xk7d7RDkK_8bILy7lVe2NJlE4y7T_", "url":"http://ip-api.com/json"}' https://4xyqiuexi7l3fcnuyuevjgclei0jhpuy.lambda-url.us-east-1.on.aws/
#應該會得到如下結果
"296 http://ip-api.com/json"

免費額度使用情況可以在 Billing > Free Tier 頁面查看。Lambda 服務每月有 400000 seconds(GB-Second,如果的函數和我的一樣用的是 128 MB,那就可以乘以 8)和 1000000 次請求。

分類
程序

在 VS Code 中調試 Django Q 任務

在 .vscode 文件夾中創建一個 lanch.json 文件, 內容如下:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Django Q",
            "type": "python",
            "request": "launch",
            "program": "${workspaceFolder}/manage.py",
            "args": [
                "qcluster"
            ],
            "django": true,
            "env": {
                "DJANGO_SETTINGS_MODULE": "YOUR_PROJECT_NAME.settings"
            },
            "gevent": true
        },
        // Your other configurations go here
    ]
}

只需修改兩個地方:program 裏的 ${workspaceFolder} 是變量,如果你項目的 manage.py 在 VS Code 項目根目錄下,那這行就不用修改。 DJANGO_SETTINGS_MODULE 這裏需要把 YOUR_PROJECT_NAME 替換成你項目的名字。然後按 Debug 按鈕就可以調試 Django Q task 了。

對於這個問題,ChatGPT 4 前前後後給出了五次錯誤配置,不得不記錄一下。

分類
程序

使用 Cloudflare Worker 中轉 HTTP 請求

Cloudflare Worker 可以方便的中轉 HTTP 請求,下面示例是我之前用過的,算是密碼保護的中轉特定請求。其中的 X_Custom_PSK 算是密碼,在 Settings > Variables 設置,這樣就只有我的程序可以請求。

addEventListener("fetch", event => {

  const psk = event.request.headers.get("X_Custom_PSK");
  if (psk === X_Custom_PSK) {
    event.respondWith(handleRequest(event.request));
  }else{
    const failed_response = new Response('Sorry, you have supplied an invalid key.', {
      status: 403,
    });
    event.respondWith(failed_response);
  }
})

async function handleRequest(request) {
  const resp = await fetch('https://domain.ltd/checkpoint/list', {
      method: 'POST',
      headers: {
          'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.3 Safari/537.36',
          'Accept': '*/*',
          'Accept-Language': 'zh-CN;q=0.8,zh;q=0.6,en-US;q=0.4,en;q=0.2',
          'Origin': 'https://domain.ltd',
          'DNT': '1',
          'Connection': 'keep-alive',
          'Referer': 'https://domain.ltd/',
          'Sec-Fetch-Dest': 'empty',
          'Sec-Fetch-Mode': 'cors',
          'Sec-Fetch-Site': 'cross-site'
      },
      body: new URLSearchParams({
          'page': '1',
          'pageSize': '10',
          'latitude': '22.5',
          'longitude': '114.0',
          'queryType': '0'
      })
  });
  return resp;
}

下面這個則是用一個 worker 代理多個網站。

addEventListener("fetch", event => {
  let url=new URL(event.request.url);
  if (event.request.url.indexOf('shaman')>-1){
      url.hostname="ft.shaman.eu.org";
  }else{
      url.hostname="www.rfi.fr";
  }  
  let request=new Request(url,event.request);
  event.respondWith(fetch(request));
});
分類
Linux 程序

Django簡易搭建上傳文件

這裡使用Django+Gunicorn+Nginx的方式簡單運行一個小型webserver,實現一個簡單的上傳文件到服務器的功能(並不生成下載鏈接)。

啟動虛擬環境,安裝django和gunicorn:

pip install Django==2.0
pip install gunicorn
#進入要放置代碼的目錄並新建項目
django-admin startproject mysite
##或者在當前目錄建立項目
#django-admin startproject mysite .
cd mysite
#新建app
python manage.py startapp polls

先建立一個表格:

#polls/forms.py
from django import forms

class UploadFileForm(forms.Form):
    title = forms.CharField(label='密碼',max_length=20,widget=forms.PasswordInput)
    file = forms.FileField(label='文件',)

修改view:

#polls/views.py
import os
import subprocess
from django.core.files.storage import FileSystemStorage
from django.conf import settings
from django.shortcuts import render
from django.http import HttpResponse
from .forms import UploadFileForm

#handle file example with file
def handle_uploaded_file(f):
    with open('/file/should/be/saved/here/target.odt', 'wb+') as destination:
        for chunk in f.chunks():
            destination.write(chunk)

#another handle file example with filename
def handle_uploaded_file2(filename):
    msg=''
    try:
        targetZipFilePath = os.path.join(settings.BASE_DIR, filename)
        cmd1=subprocess.check_call(["unzip", "-o", targetZipFilePath, "-d", "/home/fred/workspace/"])
        if cmd1==0 :
            cmd2=subprocess.check_call(["cp", "-Rf", "/home/fred/workspace/dist", "/home/fred/"])
            if cmd2==0 :
                msg="deployed successfully"
            else:
                msg="error 2"
        else:
            msg="error 1"
    except:
        msg = 'error 0'
    return msg

def upload_file(request):
    if request.method == 'POST':
        form = UploadFileForm(request.POST, request.FILES)
        if form.is_valid() and request.POST['title']=='Secret':
#            handle_uploaded_file(request.FILES['file'])
#            return HttpResponse("上傳成功")
            myfile = request.FILES['file']
            fs = FileSystemStorage()
            filename = fs.save(myfile.name, myfile)
            uploaded_file_url = fs.url(filename)
            print(uploaded_file_url)
            res = handle_uploaded_file2(uploaded_file_url)
            return HttpResponse(res)
    else:
        form = UploadFileForm()
    return render(request, 'upload.html', {'form': form})

新建一個表格的模板:

#polls/templates/upload.html
<form enctype="multipart/form-data" action="/polls/upload/" method="post">
    {% csrf_token %}
    {{ form }}
    <input type="submit" value="上傳" />
</form>

新建一個url路由表:

#polls/urls.py
from django.urls import path

from . import views

urlpatterns = [
    path('upload/', views.upload_file, name='upload_file'),
]

修改項目路由:

#mysite/urls.py
from django.contrib import admin
from django.urls import include,path

urlpatterns = [
    path('polls/', include('polls.urls')),
    path('admin/', admin.site.urls),
]

修改項目設置:

#mysite/settings.py
INSTALLED_APPS = [    
    'polls.apps.PollsConfig',
#    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
#    'django.contrib.messages',
#    'django.contrib.staticfiles',
]
LANGUAGE_CODE = 'zh-Hant'
TIME_ZONE = 'Asia/Taipei'

然後在項目目錄(最上層)運行gunicorn就可以訪問了:

gunicorn mysite.wsgi --bind 127.0.0.1:3040

nginx中增加如下server即可在外網訪問了(鏈接應該是http://YourPublicIP:8081/polls/upload/):

server {
        listen 8081;
        server_name 127.0.0.1;
        charset utf-8;
        keepalive_timeout 60s;
        #access_log logs/django2a.access.log combined if=$loggable;
        
        location / {
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_redirect off;
    
            if (!-f $request_filename) {
                proxy_pass http://127.0.0.1:3040;
                break;
            }
    }
}

CentOS6+Django2+MySql

yum install MySQL-python
pip install mysqlclient
#如果import MySQLdb提示無libmysqlclient.so.18
#則建立軟連接如下
ln -s /usr/local/mysql/lib/libmysqlclient.so.18 /usr/lib64/libmysqlclient.so.18
分類
程序

把媒體中地理位置標籤從 WGS-84 轉換成 GCJ-02

手機拍攝的照片地理位置默認是 WGS-84 系統,但是谷歌地圖使用的卻是 GCJ-02 系統,這就會出現偏移。爲了展示正確的位置,可以通過把照片的地理位置標籤轉換成 GCJ-02 來實現。需要用到 Phil Harvey開發的 ExifTool 和 Avatar sshuair 開發的 coord-convert。前者下載exiftool.exe 到本地即可,後者可以通過 pip 安裝。

# -*- coding: utf-8 -*-

#pip install coord-convert

#Usage:
#Drage a single photo or directory with photos to this script
# OR
#python wgs2gcj.py D:\my.JPG
#Results will be wrote to gps.log at the same dir with this script

from coord_convert.transform import wgs2gcj
import subprocess,json,sys,os,time,datetime
#You may set your exiftool file here
exiftool_path = "D:\\Program Files\\portable\\exiftool.exe"

work_dir = os.getcwd()
def log42(logFile,logText):
	ts = int(time.time())
	dt = datetime.datetime.fromtimestamp(ts).strftime('%Y-%m-%d %H:%M:%S')
	
	log_file = os.path.join(work_dir,logFile+".log")
	with open(log_file, "a") as myfile:
		myfile.write(dt+" "+logText+"\n")
		print(logText)
		
		
if (exiftool_path == ""):
	exiftool_path = os.path.join(work_dir,'exiftool.exe')
if not os.path.exists(exiftool_path):
	log42("gps","Error: exiftool.exe not found neither in script directory nor in the script setting \"exiftool_path\"")
	exit()
	
file_paths_input = sys.argv[1:]
file_paths = []

for file_path_input in file_paths_input:
	if os.path.isfile(file_path_input):
		file_paths.append(file_path_input)
	elif os.path.isdir(file_path_input):
		for root,ds,fs in os.walk(file_path_input):
			for ff in fs:
				full_path = os.path.join(root,ff)
				file_paths.append(full_path)

for f in file_paths:
	output = subprocess.check_output('"'+exiftool_path+'" -struct -j -a -n "-gps*" '+f, shell=True)
	data = json.loads(output)
	if 'GPSMapDatum' in data[0]:
		log42("gps",data[0]['SourceFile']+':'+str(data[0]['GPSMapDatum'])+' '+str(data[0]['GPSLatitude'])+' '+str(data[0]['GPSLongitude']))
		if(data[0]['GPSMapDatum']=='WGS-84'):
			gcj_lon, gcj_lat = wgs2gcj(data[0]['GPSLongitude'], data[0]['GPSLatitude'])
			log42("gps",str(gcj_lat)+' '+str(gcj_lon))
			if (gcj_lat==data[0]['GPSLatitude']):
				log42("gps",'not in China')
			else:
				output = subprocess.check_output('"'+exiftool_path+'" -overwrite_original -exif:gpsmapdatum=GCJ-02 -exif:gpslatitude='+str(gcj_lat)+' -exif:gpslongitude='+str(gcj_lon)+' '+f, shell=True)
				log42("gps",output.decode())
		else:			
			log42("gps","not a wgs84 location")
	else:
		log42("gps","no gps data found")
		
log42("gps","task finished")

另請參考:使用ExifTool處理文件元數據wgs-84-gcj-02-經緯度在線轉換

分類
程序

PHP 使用 Nominatim 的逆地理編碼

之前一直使用谷歌的逆地理編碼服務,多年來運行良好。但是谷歌畢竟是一家巨大的商業公司,所以我現在改用 Nominatim

Nominatim(來自拉丁語,“按名稱”)是一個通過名稱和地址搜索開放街圖數據,根據開放街圖點來生成合成地址(逆地理編碼)的工具。
<?php
header("Access-Control-Allow-Origin: *");
header('Content-type: text/json');
header("Cache-Control: no-cache, must-revalidate");

$lat=$_POST["lat"];
$lng=$_POST["lng"];
$lang=$_POST["lang"];

$url = "https://nominatim.openstreetmap.org/reverse?format=geojson&lat=".$lat."&lon=".$lng."&accept-language=".$lang;

$ch = curl_init(); 
//set your own agent name
$agent = 'location/0.2(ft.shaman.eu.org)';
curl_setopt ($ch, CURLOPT_URL, $url); 
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1); 
curl_setopt ($ch, CURLOPT_CONNECTTIMEOUT,20); 
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
$content = curl_exec($ch); 
curl_close($ch);

$content = json_decode($content);
$resultArray = array();
foreach($content->features[0]->properties->address as $key => $value) {
    //remove country_code and postcode
    if ($key=='country_code' or $key=='postcode'){            
    }else{
        array_push($resultArray,$value);
    }
}
echo implode(",",$resultArray);
?>	
curl -d 'lat=22.57776&lng=113.94849&lang=zh-TW' https://your.api.domain/api.php
西丽华昌大厦,西丽南路,松坪村,南山区,西丽街道,广东省,中国
分類
程序

使用python同步本地時間

服務器超售到不行,cpu時間隔一段時間就滿好多。嘗試配置了標準了自動的時間同步,效果不理想。於是用python調用worldtimeapi.org的api配合crontab來修正系統時間。

import subprocess,requests,time

def main():
    try:
        response = requests.request("GET", "http://worldtimeapi.org/api/timezone/Asia/Hong_Kong")
        timeJson=response.json()
        timeAbs = abs(timeJson['unixtime']-time.time())
        if timeAbs > 15 :
            subprocess.call(['date','+%T','-s',timeJson['datetime'][11:19]])
            print("time synced")
        else:
            print(timeAbs)
            
    except:
        print("net error")
    
if __name__ == '__main__':
    main()
#添加crontab任務
crontab -e
#每兩小時檢查一次
0 */2 * * * /usr/local/bin/python3 /home/42/time.py >> /home/42/t.log 2>&1